Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataloading utils #85

Draft
wants to merge 22 commits into
base: master
Choose a base branch
from
Draft

Conversation

daler3
Copy link
Contributor

@daler3 daler3 commented Dec 22, 2020

Description

Work in progress pull request for dataloading utils, dataloaders and datasets.

Affected Dependencies

Currently using PySyft 2.0. To be changed to not using PySyft at all, or eventually PySyft 3.0

How has this been tested?

Manually, unit and integration tests to be properly added

Checklist

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@daler3
Copy link
Contributor Author

daler3 commented Dec 22, 2020

To Resolve #81

import datasets


"""I think this is not needed anymore"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean we don't need the partitioned dataloader?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I mean that the default pytorch dataloader in PyTorch works, so we do not need a custom one (for how it is done now). See the notebook for an example.

self.values = torch.Tensor(values) if is_labels else torch.stack(values)

self.worker_id = None
if worker_id != None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that can simplify to if worker_id:

fmt_str = "FederatedDataset\n"
fmt_str += f" Distributed accross: {', '.join(str(x) for x in self.workers)}\n"
fmt_str += f" Number of datapoints: {self.__len__()}\n"
return fmt_str
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline at the end of the file

self.dataset = dataset #It can also be None, and then it would be only computational
self.model = model

self.level = level if level >= 0 else 0 #it should start from zero, otherwise throw error #TODO: implement error throwing
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simplify to max(level, 0)

This code is meant to be used with dual-headed Neural Networks, where there are a bunch of different workers,
which agrees on the labels, and there is a server with the labels only.
Code built upon:
- Abbas Ismail's (@abbas5253) work on dual-headed NN. In particular, check Configuration 1:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this PR require abbas' PR to be merged?

the third the index, which is to keep track of the same data point.
"""

if worker_list == None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if worker_list: or if worker_list is None:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants