https://github.com/cdcgov/nedss-dataingestion - v7.11.0

What's Changed

• Fix/match identification by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/439
• Change default sort to identification date by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/442
• [CNDE-2601] Updated config to tun swagger off in prod by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/441
• persist pre-merge patient data to support unmerge by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/438
• Fix missing columns in person_merge during NBS 7 merge by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/444
• Merge - Investigation/HIV Permission issues by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/445
• Add sort to merge review query by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/446
• [CNDE-2601] Updated Postman Collection by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/447
• Added README for postman collection by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/449
• Merge review - Sort investigations by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/450
• Merge Sync - Remove deleted records from merge queue by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/451
• Update Github action by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/448
• Bump data-ingestion-service version to 2.0.0-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/453
• [CNDE-3043] Updated ELR Rhapsody Route by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/452
• Updated DI generated phdc xml to match with Rhapsody. by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/454
• Bump data-ingestion-service version to 2.0.1-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/457
• Bump data-ingestion-service version to 2.0.2-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/458
• Persist user and timestamp when 'No merge' is selected by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/456
• Bump data-ingestion-service version to 2.0.3-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/459
• Bump data-ingestion-service version to 2.0.4-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/460
• Bump data-ingestion-service version to 2.0.5-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/461
• Bump data-ingestion-service version to 7.11.0-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/464
• Bump data-processing-service version to 7.11.0-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/465
• Bump data-ingestion-service version to 7.11.0-SNAPSHOT by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/466
• Bump data-processing-service version to 7.11.0-SNAPSHOT-0001 by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/467
• Bump data-processing-service version to 7.11.0-SNAPSHOT-0002 by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/468
• Ensure race data matches a valid race by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/462
• Make includemultiplematches configurable by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/470
• Upgrade Record Linker container to v25.9.0 by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/469
• reenabled prometheus endpoint by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/471
• Merge patients - fix failures by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/472
• Promeheus by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/473
• Fix for pregnancy status observation and county name issue by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/474
• Merge - review schema update by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/475
• Update - add raw id as comment in xml payload by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/463
• DP - fixed CVE by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/479
• Bump data-processing-service version to 7.11.0-SNAPSHOT-0001 by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/480
• Bump data-processing-service version to 7.11.0-SNAPSHOT-0003 by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/481
• API: Merge Queue by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/478
• DP CVE by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/482
• Bump data-processing-service version to 7.11.0-SNAPSHOT-0004 by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/483
• Move GET above DELETE by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/485
• RTI bugfixes - Multi OBR, EDX log related by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/477
• Bump data-processing-service version to 7.11.0-SNAPSHOT-0005 by @ndduc-bot in https://github.com/CDCgov/NEDSS-DataIngestion/pull/486
• Added Swagger enable flag for RTI by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/487
• Skip Self Match: Batch Job Merge by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/488
• persist updates to _hist tables during patient merges by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/476

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.10.3...v7.11.0

- Java
Published by achapm 11 months ago

https://github.com/cdcgov/nedss-dataingestion - v7.10.3

What's Changed

• Deduplication - UI API authentication by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/427
• Additional Support for RTI by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/426
• Upgrade to spring boot 3.5.3 by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/430
• DI HL7 batch feature flag by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/429
• Merge API updates by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/431
• DI obr splitter fix by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/433
• Merge - Transaction rollback by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/435
• DI obr splitter fix by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/436
• Cnde 3032 - DP bugfix to support OBR Splitter by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/432
• Fix seed query for multiple pages by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/437
• ECR bug fixes from NIS-76 by @sralla-eq in https://github.com/CDCgov/NEDSS-DataIngestion/pull/440

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.10.2...v7.10.3

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.10.2

What's Changed

• [CNDE-2925] ECR data quality fix and new logic added for assigned author section by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/425
• [CNDE-2920] Updated eCR controller and service file - security fix by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/428

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.10.1...v7.10.2

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.10.1

What's Changed

• Sort merge candidate table by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/372
• Fix count query by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/374
• process all previous day records in addition to configured limits by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/367
• Debezium move by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/375
• Cnd 303 sync updates by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/377
• PDF Header fix by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/378
• Adjust endpoints for pdf/csv export by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/380
• Update batch job to store NBS person ID in match_candidates table by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/379
• Add API to remove (keep separate) patients from merge list by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/381
• Support UI admin comments section by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/382
• normalize match candidate structure with new join table by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/383
• Cnd 283 repeating sections by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/384
• [CNDE-2825] Added ecr lookup tables into liquibase by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/385
• Merge details update by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/386
• Improve fault tolerance in sync job by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/387
• [CNDE-2825] Syntax fix for ecr lookup liquibase scripts by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/389
• Cnd 285 ethnicity by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/388
• Patient merge - sex and birth by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/390
• [CNDE-2847] HOTFIX- Code value fix for question identifier by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/391
• Mortality mapping by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/392
• CVE: update tomcat by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/395
• Merge review - General info by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/396
• Investigation by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/397
• Implement initial patient record merge for admin comments and person by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/393
• Cnd 346 patient local by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/398
• Fix mortality to not require location entry by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/399
• Clean up remove from merge endpoints by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/401
• Implement merging of selected person names by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/400
• Sync PAT only by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/402
• Implement address merge logic for patient merge API by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/404
• Implement phone & emai merge logic for patient merge API by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/405
• Record Linker v25.8.0 by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/407
• Implement Identification record merge into surviving patient by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/406
• RIT Performance Improvement by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/373
• [CNDE-2847] eICR NOT_MAPPED bug fix by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/410
• UNIT TEST by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/408
• Update sonar.yaml by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/413
• Revert "Update sonar.yaml" by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/415
• Code Coverage by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/412
• implement race merging for patient merge process by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/409
• Clean up by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/416
• Sync - handle deletions by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/417
• enabled web app by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/418
• Fix UID by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/419
• Dlt update 06192025 by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/420
• Resolved CVE by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/422
• Implement Ethnicity merge logic for patient merge API by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/421
• Prevent duplicate container startup by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/423
• Implement Patient-data merge logic for patient merge API by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/424

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.9.3...v7.10.1

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.10.0

What's Changed

• [CNDE-2847] HOTFIX- Code value fix for question identifier https://github.com/CDCgov/NEDSS-DataIngestion/pull/391
• CVE: update tomcat https://github.com/CDCgov/NEDSS-DataIngestion/pull/395

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/release-7.9.3...release-7.10.0

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.9.3.1

What's Changed

• [CNDE-2847] HOTFIX- Code value fix for question identifier https://github.com/CDCgov/NEDSS-DataIngestion/pull/391
• CVE: update tomcat https://github.com/CDCgov/NEDSS-DataIngestion/pull/395

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/release-7.9.3...release-7.9.3.1

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.9.3

What's Changed

• Add API to retrieve merge candidates by patient ID. by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/358
• CND 294: Export CSV for Matches Requiring Review by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/360
• Add an API to retrieve merge candidate extra data by patient ID. by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/361
• Record Linker 25.5.0 by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/362
• Cnd 298 batch limit by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/364
• Deduplication - Threshold update by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/365
• CND-295: Export Merge Candidate List as PDF by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/366
• Resolved the issue on DI sftp that occurs when a file name contains dots. by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/369
• CVE fix. upgraded spring boot version to 3.4.5 by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/371

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.9.2...v7.9.3

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.9.2

New Features & Enhancements:

• The Data Ingestion service now supports retrieving ELR files over SFTP, expanding the options for secure data transfer.
• Administrators can now configure file paths and specify file types for ELR retrieval, providing greater flexibility and control over data ingestion sources.
• The system has been enhanced to ingest ELR files from multiple folders within the same parent directory, improving coverage and operational efficiency.
• The Data Ingestion API error details endpoint now accepts start date and end date parameters, enabling users to retrieve error records for a specific date range.

What's Changed

• Add medicare number to identifications by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/344
• DI sftp support by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/343
• Branch for daterange, elr formatting, and validation implementation by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/320
• DI sftp custommapper fix by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/347
• Remove telecom by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/348
• Fix on getting the status from DI sftp module. by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/349
• sftp env params name change. by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/350
• Update seeding to resume from the last processed record by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/345
• Implement API to merge patients. by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/324
• Retrieve merge candidate list for review by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/353
• DI bug fixes by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/354
• Sgz pipeline updates by @szamfir in https://github.com/CDCgov/NEDSS-DataIngestion/pull/355
• update sonar token handling by @szamfir in https://github.com/CDCgov/NEDSS-DataIngestion/pull/356
• Support paging of merge candidate list by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/357
• Updated Kafka version and fix for formatting the elr specimen col date by @ssathiah in https://github.com/CDCgov/NEDSS-DataIngestion/pull/359

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.9.1...v7.9.2

- Java
Published by achapm about 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.9.1

What's Changed

• Update base path to /nbs/api/deduplication. Remove /api from controllers by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/331
• Add health endpoint. Remove unnecessary semicolon by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/334
• [CNDE-2429] Added healthchecks for DI and SRTE services by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/335
• Lock record linker version by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/337
• [CNDE-2485] Added config to pull tag from git to update version by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/336
• DP: expose health check by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/332
• Cnde 2458 - ECR Liquibase by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/330
• Revert "[CNDE-2485] Added config to pull tag from git to update version" by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/338
• LIQUIBASAE: update by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/339
• SRTE: fix srte actuator by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/340
• Set deserialization for enums by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/341
• Call update helm for new container by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/342
• DI and DP: vulnerability by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/346

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.9.0...v7.9.1

- Java
Published by achapm over 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.9.0

What's Changed

• Rti fix by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/281
• Improve performance of MPI seed process. by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/280
• Deduplication container workflow by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/283
• RTI: added kafka error check for step 2 and step 3 -- addressing kafka reb… by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/282
• Remove warnings for generated files by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/287
• [CNDE-2115] New endpoint to pull all ingested reports by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/286
• Update model for RecordLinker v25.1.0 by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/289
• add workflow for srte data service by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/293
• Record linker container by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/292
• Check for expanded persons to be active by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/295
• Fix spacing in workflow by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/294
• Adding batch job for De-duplication. by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/299
• ECR Ingestion query limit by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/301
• Fix annotation. add MS to timing by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/303
• CND-170 by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/302
• Fix toString and exception test by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/304
• DI - Setting up support for multi partition and Kafka concurrency by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/300
• RTI: changing elrunprocess to dpelr_unprocessed by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/306
• CND-193-Default-DIBBS-Configuration-Initial-State by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/305
• cnd-195-import api and unit test by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/309
• CND-194 export api and unit test by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/308
• Update record linker 25.2.0 by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/310
• CND-227: Export/Import Configuration with Data Elements by @snehaasjournal in https://github.com/CDCgov/NEDSS-DataIngestion/pull/312
• Sync job development by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/311
• DI - Fix payload size by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/313
• [CNDE-2041] DLT update to retry failed messages by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/315
• API to retrieve patient groups by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/314
• Add API to remove a merge group and mark as 'No Merge' by @sizatenquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/316
• RTI: Changes for performance by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/291
• DP: fixed DP unit test by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/318
• Deduplication API - configuration management by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/322
• CND 209 export api by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/323

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.8.1...v7.9.0

- Java
Published by achapm over 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.8.1

Bug Fixes

• Correct timestamp is displayed on the NBS UI once ELR is ingested through the API

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.8.0...v7.8.1

- Java
Published by achapm over 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.8.0

What's Changed

• CNDE-1882 update security by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/241
• [CNDE-1895] NBS_interface column not populating bug fix by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/243
• increase heap memory by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/244
• update heap memory usage by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/247
• Add deduplication folder with database container" by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/246
• Local containers by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/248
• added full edx detail object for status by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/245
• using differnt xml package to avoid vulnerability scan by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/251
• Deduplication - MPI Seed process by @mpeels in https://github.com/CDCgov/NEDSS-DataIngestion/pull/250
• Move In Memory Object to a separate service (Addressing out of memory issue with DP) by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/249
• Fixing di build by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/253

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.7.1...v7.8.0

- Java
Published by achapm over 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.7.2

New Features

• Added support for Lab 100 and Lab 101 datamarts and dimensions to the near real time reporting pipeline.

What's Changed

• RTI: update log switch logic to return the correct message by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/233
• Reducing High Code Smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/234
• Cnde 1565 3 by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/236
• Cnde 1565 4 by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/237
• clean up remaining code smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/238
• CNDE-1565 - Clean up code smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/239
• Final clean on current code smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/240

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.7.0...v7.7.2

- Java
Published by achapm over 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.7.1

New Features

• The data Ingestion ELR pipeline now has the option to use Near Real-Time Ingestion (NRTI) to enable event-based ingestion, enhancing both speed and scalability.

What's Changed

• RTI: update log switch logic to return the correct message by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/233
• Reducing High Code Smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/234
• Cnde 1565 3 by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/236
• Cnde 1565 4 by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/237
• clean up remaining code smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/238
• CNDE-1565 - Clean up code smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/239
• Final clean on current code smell by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/240

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.7.0...v7.7.1

- Java
Published by achapm over 1 year ago

https://github.com/cdcgov/nedss-dataingestion - v7.7.0

New Features

• The Data Ingestion API now has the ability to ingest ECR PHDC XML data. Rhapsody can call this API to send ECR and ELR xml in PHDC format to be processed and imported into the ODSE tables. More documentation has also been added covering configuration and workflow for the Data Ingestion API including Rhapsody Route Integration.
• A new on-premises Data Sync Service that synchronizes data from the cloud NBS database to an on-premises database. It offers flexible scheduling and supports multiple sync options such as SQL DB sync, S3 Bucket and Local Directory.

What's Changed

• update edx log so it not adding duplicated data by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/227
• supporting multiple uid for status endpoint by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/229
• added sub path by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/228
• encoding payload to base64 by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/230
• Update openAPI config to take new sub path into account by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/231
• [CNDE-1774] Adding sql script for ECR local table creation by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/232

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.6.2...v7.7.0

- Java
Published by achapm almost 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.6.2

New Features Added to Data Ingestion and Near Real Time Reporting

• A new API has been added enabling the ingestion of ELR PHDC XML data without directly linking Rhapsody to database. Rhapsody calls this API to send PHDC.xml to be processed by NBS ELR importer process into ODSE tables. In the next release more documentation will be added covering configuration and workflow for the new API. In this release the Swagger documentation will be available here:
  • https://<>/swagger-ui/index.html
  • Sample swagger documentation here: https://dataingestion.dts1.nbspreview.com/swagger-ui/index.html
• Added Public Health Case Fact (PHCF) datamart to the near Real Time Reporting pipeline. This adds support for standard reports that reference the PHCF datamart within the ODSE to get data in near real time. This brings the total number of datamarts that have been added to the pipeline for the near real time reporting to 2 as follows:
  
  • Hepatitis (HEP)
    
  • Public Health Case Fact (PHCF)
    

What's Changed

• [CNDIT-1758] CDA mapper changes for ECR by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/225
• Pull request template added by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/226

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.6.1...v7.6.2

- Java
Published by achapm almost 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.6.1

Bug Fixes and Enhancements

• Upgraded to Java 21
• Addressed security issues

What's Changed

• [CNDIT-1735] Removed actuator endpoint from auth whitelist by @rshanmugam-enquizit in https://github.com/CDCgov/NEDSS-DataIngestion/pull/217
• Dev dn fix transaction issue and null exception by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/219
• upgrading java to 21 - and resolve dependencies vulnerability by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/221
• added java 21 for base image for DP by @ndduc01 in https://github.com/CDCgov/NEDSS-DataIngestion/pull/224

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.6.0...v7.6.1

- Java
Published by achapm almost 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.6.0

New Features

• Completed a new API that allows systems hosted on-prem to retrieve data from a cloud-hosted NBS database, with an API endpoint service and without interrupting the existing Outbound NNDSS route and message transmission systems hosted On-Prem at Jurisdictions.
• Adds near real time reporting to NBS, starting with the Hepatitis Datamart. Near real time reporting replaces the existing batch ETL process.

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.5.3...v7.6.0

- Java
Published by achapm almost 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.5.3

Updates and Enhancements

• Includes infrastructure changes for an upcoming API that allows on-prem systems to make connections to the database in the NBS Cloud. Documentation will be included in the upcoming v7.6.0 release.
• Our data team completed foundational work and testing for near real-time reporting service in the NBS cloud. Our upcoming v7.6.0 release will include support for the Hepatitis Datamart. We will add documentation and support for other Datamarts in future releases.

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.5.0...v7.5.3

- Java
Published by achapm almost 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.5.0

New Features and Enhancements

• We made significant improvements to the data ingestion testing and documentation using Swagger API tools.
• Updated the data ingestion service to follow REST API naming conversions and improved security by limiting access.
• NBS 7.5 has been updated to support the latest NBS 6.0.16 release

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.4.0...v7.5.0

- Java
Published by achapm almost 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.4.0

New Features

• The data Ingestion API now supports HL7 ELR text files uploaded via SFTP.

Bug Fixes and Enhancements

• Fixed mislabeling on data validation where 'MSH-6' was incorrectly cited instead of 'MSH-7' for date and time errors.
• Error messages accurately reflect the correct field identifiers to enhance clarity and diagnostics during data ingestion process.
• Improved validation to accurately parse timestamps in the YYYYMMDDHHMMSS format which now includes seconds to ensure successful HL7 validation and XML record creation.
• Enhanced the data validation process to specifically check for mandatory fields within HL7 earlier in the data ingestion service. This update prevents errors associated with mandatory fields from progressing to the batch job stage, ensuring a more efficient and reliable data processing workflow.
• Resolved an out-of-memory issue by removing faulty code. Users can now post any number of ELRs with improved reliability during high-volume ELR ingestion.

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.3.0...v7.4.0

- Java
Published by achapm about 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.3.0

Bug Fixes

• Enhanced CLI input validation now prompts users to enter positive integers and handles invalid input correctly, while decimal inputs are rounded to the nearest integer for consistency.
• Enhanced CLI error messaging for invalid ID input during status check requests. Users now receive a clearer notification when entering incorrect or dummy IDs, improving user experience and error handling.
• Improved error messaging for API authentication headers. Error messages now indicate which header values are invalid, facilitating quicker identification and resolution of authentication issues.

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.2.1...v7.3.0

- Java
Published by achapm over 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.2.1

New Features and Enhancements

• AuthN/AuthZ service V2. Added secure service-to-service authentication onto the Data Ingestion Pipeline using Keycloak Identity Provider (IdP)
• Added ability to use CLI commands on Windows and Linux to support all users, instead of only supporting Macs. CLI commands allow users to perform status checks on imported Electronic Lab Report, verify Electronic Lab Report format of a record, and to review errors during the ingestion process when using Data Ingestion pipeline.
• Includes Keycloak deployment with support for modern standards-based SSO that works out of the box and integrates with existing widely adopted identity providers.

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.2.0...v7.2.1

- Java
Published by achapm over 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.2.0

New Features and Enhancements

• Added ReportStream setup and configuration with ability to send HL7 ELR from ReportStream directly to NBS.
• Added onboarding flows to data ingestion CLI, including commands to status check imported ELR, verify ELR format of a record, and to review errors during the ingestion process when using Data Ingestion pipeline.
• Added error messaging by adding error handling and retry logic to Data Ingestion pipeline.

Bug Fixes

• Users are no-longer allowed to post HL7 message when "msgType" and "validationActive" headers are invalid or missing values.
• Removed erroneously hardcoded values within the nbs_interface table and replaced with correctly referenced value.

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/compare/v7.1.0...v7.2.0

- Java
Published by achapm over 2 years ago

https://github.com/cdcgov/nedss-dataingestion - v7.1.0

What's Changed

This is the first release of the Data Ingestion Service which provides necessary foundational pieces to track and route Electronic Lab Reports data flowing into NBS and lays the groundwork to provide additional ingestion options for NBS. The following features for the Data Ingestion Service are included in this first release:

• Accepts a variety of electronic lab reports using different versions and fields
• Saves all incoming messages for auditing, debugging, and disaster recovery and you can configure how long messages are saved
• Verifies all input data (HL7) against standard rules and provides an error when rejected
• Supports both positive and negative lab results with the appropriate code
• Provides both syntactic and semantic validations
• Remove unwanted data such as extraneous logs using filters
• Provides duplication checks to see if the data has made it through the system before
• Includes error handling and logging for both business data and operation data for situational awareness
• Supports traffic and system health monitoring

Full Changelog: https://github.com/CDCgov/NEDSS-DataIngestion/commits/v7.1.0

- Java
Published by achapm over 2 years ago