Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor CRIS import script to use S3 and log activities #1445

Merged
merged 52 commits into from
May 14, 2024
Merged

Refactor CRIS import script to use S3 and log activities #1445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
40a6499
some handy tooling
frankhereford May 1, 2024
8d84187
rough-in state tracking and s3 integration
frankhereford May 1, 2024
8893115
default uploads db
frankhereford May 1, 2024
566c670
import botocore
frankhereford May 1, 2024
b9c690a
new env variable
frankhereford May 1, 2024
d9c00ca
extra bit of state for the operation
frankhereford May 1, 2024
597ce76
new downloading from s3 mechanics
frankhereford May 1, 2024
4d35d23
rough in uploading db
frankhereford May 2, 2024
530fee1
exclude the directory of PDFs, which exists even if empty
frankhereford May 6, 2024
ed2f7c3
remove old functions being replaced
frankhereford May 6, 2024
61d3e5f
get out the ID so we can update it later if we succeed
frankhereford May 6, 2024
f499674
mark completion in the sqlite db
frankhereford May 6, 2024
ee6fdf8
no tracking binary files, even if tiny
frankhereford May 6, 2024
3ae67ec
record the schema itself instead
frankhereford May 6, 2024
3ed958c
let's not make that mistake again
frankhereford May 6, 2024
d8653a4
tag-along change - remove version designation from docker compose file
frankhereford May 6, 2024
ca4887c
add migration to put the table in pg
frankhereford May 7, 2024
e9f133a
get pg connection instead of sqlite
frankhereford May 7, 2024
19965b4
kick this whole block in an indent
frankhereford May 7, 2024
c6ae9be
don't include note column per conversation
frankhereford May 7, 2024
4b62460
convert function to use pg
frankhereford May 7, 2024
8007118
port this function to pg as well
frankhereford May 7, 2024
a043cdb
and one last function being brought over to PG
frankhereford May 7, 2024
ffa7b12
refactor for psycopg
frankhereford May 8, 2024
9ed0e45
move file after attempted
frankhereford May 8, 2024
1102d10
more psycopg refactoring
frankhereford May 8, 2024
dd0e5ca
reinstate deployment environment directories
frankhereford May 8, 2024
b1244a3
drop this errant column
frankhereford May 8, 2024
9148140
Merge branch 'master' into use-s3-for-cris-delivery
frankhereford May 8, 2024
31fda55
don't need this anymore
frankhereford May 8, 2024
9041064
don't need this anymore
frankhereford May 8, 2024
2642593
remove unused global now
frankhereford May 8, 2024
df60832
throw exception if no work to do
frankhereford May 8, 2024
abf7f06
seeya, wouldn't want to be ya
frankhereford May 8, 2024
2baf51c
rename symbol
frankhereford May 8, 2024
3f84a16
remove unneeded import
frankhereford May 8, 2024
01ec3ae
check for file download success and that it's a real zip file
frankhereford May 8, 2024
0523abd
really use the log table as a log
frankhereford May 8, 2024
cffe3c0
update sql to new table columns
frankhereford May 8, 2024
7407767
woops, shouldn't have committed something so temporary
frankhereford May 8, 2024
3d17545
quell noise
frankhereford May 8, 2024
ea77275
how has this not been a thing?
frankhereford May 9, 2024
cdffadf
fixme notation
frankhereford May 9, 2024
c48a934
destructure the tuple properly
frankhereford May 9, 2024
c0a3e64
move the extract around
frankhereford May 9, 2024
2a7e233
fix moving after you're done
frankhereford May 9, 2024
07082c6
remove this old code
frankhereford May 9, 2024
1c65ffe
pass additional required info in main()
frankhereford May 9, 2024
b9ce5eb
outdent db handle handling
frankhereford May 9, 2024
e2ca36e
add some column comments to the migration
frankhereford May 9, 2024
92f2161
tag-along field name correction, h/t @johnclary
frankhereford May 9, 2024
925e97b
bump patch version for JS apps
frankhereford May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion atd-etl/cris_import/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ RUN ln -snf /usr/share/zoneinfo/America/Chicago /etc/localtime && echo America/C
RUN apt-get update && apt-get install -y tzdata

RUN apt-get -y upgrade
RUN apt-get install -y aptitude magic-wormhole vim black python3-pip
RUN apt-get install -y aptitude magic-wormhole vim black python3-pip libmagic1 file
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using libmagic to check that the zips are zips through inspection of the binary file.


# install zip archive tool & pgloader for CRIS export use
RUN apt-get install -y p7zip-full pgloader rsync
Expand Down
Loading
Loading