Publishing Data Assets for Multi-Stage Computation
Federated Compute enables algorithms to run across distributed datasets simultaneously and aggregates the results into a single final output
For a Federated Compute scenario to work, you'll need to publish at least 4 different data assets:
- An Algorithm for Aggregation: the algorithm on the second stage of computation, responsible for combining the results into a unified output.
- An Algorithm for Federation: the algorithm on the first stage of the computation, responsible for running across distributed datasets.
- The Dataset(s) (Compute type): the assets in which the Algorithm for Federation will run on.
- Any placeholder dataset to define the location of computation (Compute type).
Step 1: Publish the Algorithm for Aggregation
- Log in to the marketplace.
- From the left navigation bar, go to Create > Algorithm.
- Under Algorithm Type, select 'Aggregate'.
- Key in all other required information for publishing an algorithm. For more details, please refer to Publishing an Algorithm.
Step 2: Publish the Algorithm for Federation
- Log in to the marketplace.
- From the left navigation bar, go to Create > Algorithm.
- Under Algorithm Type, select 'Federated'.
- Under Define your aggregate algorithm, select the Aggregate Algorithm that you want to use or just published from Step 1.
- Key in all other required information for publishing an algorithm. For more details, please refer to Publishing an Algorithm.
Step 3: Publish a Dataset (Compute)
- Log in to the marketplace.
- From the left navigation bar, go to Create > Dataset.
- Under the Access Type, click on 'Compute'.
- Do NOT enable Aggregate only. Keep it as 'Disable'.
- Under Compatible Algorithm(s), select the Federated Algorithm that you want to use or just published from Step 2.
- Key in all other required information for publishing a compute-type dataset. For more details, please refer to Publishing a Dataset for Compute-to-Data.
- Repeat the same steps to publish additional datasets for your usage.
Step 4: Publish a Placeholder Dataset
- Log in to the marketplace.
- From the left navigation bar, go to Create > Dataset.
- Under the Access Type, click on 'Compute'.
- Enable Aggregate only option.
- Make sure to select the Edge Node you wish to use to run the computation during the aggregation stage.
- Under Compatible Algorithm(s), select the Aggregate Algorithm that you want to use or just published from Step 1.
- Key in all other required information for publishing a compute-type dataset. For more details, please refer to Publishing a Dataset for Compute-to-Data.
Having this 'Placeholder' dataset helps the Aggregated algorithm to know where to run the computation during the aggregation step of the multi-stage computing.