Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Field-Level Metadata Removed #101

Open
ers81239 opened this issue Jun 30, 2021 · 0 comments · May be fixed by #102
Open

Field-Level Metadata Removed #101

ers81239 opened this issue Jun 30, 2021 · 0 comments · May be fixed by #102
Labels
bug Something isn't working

Comments

@ers81239
Copy link

ers81239 commented Jun 30, 2021

Describe the bug
Field level metadata is removed from the catalog when discovery is performed

To Reproduce
Steps to reproduce the behavior:

  1. Provide field-level metadata in the catalog, for example:
{
          'breadcrumb': ['properties', 'char_name'],
          'metadata': {
	          'arbitrary_field_metadata': 'should be preserved'
          }
}
  1. Run the tap

This affects at least Meltano users who use the meltano select feature to populate the selected metadatum to indicate whether fields should be copied or not. In that case, the selected field is being stripped and therefore ignored and all fields get copied regardless.

Expected behavior
Field-level metadata should be preserved through the stream_utils.refresh_streams_schema() with only discoverable metadata being updated/added.

Your environment

  • Version of tap: master
  • Version of python: 3.8

Additional context
I'm using the tap with meltano and setting a field metadatum: selected: True, however this is stripped from the metadata and in turn not honored by the tap.

@ers81239 ers81239 added the bug Something isn't working label Jun 30, 2021
ers81239 added a commit to ers81239/pipelinewise-tap-postgres that referenced this issue Jun 30, 2021
@ers81239 ers81239 linked a pull request Jun 30, 2021 that will close this issue
7 tasks
@deanmorin deanmorin mentioned this issue Oct 7, 2021
13 tasks
ers81239 added a commit to ers81239/pipelinewise-tap-postgres that referenced this issue Oct 27, 2021
astrojuanlu added a commit to astrojuanlu/pipelinewise-tap-postgres that referenced this issue May 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant