Draft: Partition parquet dataset for sync with s5cmd

For large datasets, we or other users may want to sync large sets of files to external storage such as LTS. This merge request adds some scripts to split a piece of an entire parquet dataset into a number of partitions with sync commands for use with s5cmd. Each partition is synced in an array task. The current setting has 10 tasks running at once to try and minimize overall impact on the filesystem

prep-parquet-for-s5cmd/fpart-db.py

@@ -34,16 +34,12 @@ def main():
     ddf = dd.read_parquet(input_parquet)
     ddf = ddf.loc[ddf['path'].str.startswith(filter)].sort_values('path')
 
-
-    ddf = ddf.loc[~ddf['mode'].str.startswith('d')].reset_index(drop=True)
-
-
-    ddf['group'] = np.floor(ddf.index/split_count).astype(int) + 1
+    ddf = ddf.loc[~ddf['mode'].str.startswith('d')]
 
     ddf['cmd'] = ddf['path'].map(lambda x: create_sync_cmd(x, filter=filter, dest=dest), meta=str)
-
-
-    df = ddf[['group','cmd']].compute()
+    
+    df = ddf[['cmd']].compute().reset_index(drop=True)
+    df['group'] = np.floor(df.index/split_count).astype(int) + 1
 
 
     os.makedirs(part_dir,exist_ok=True)