Skip to content
Snippets Groups Projects
Commit 7fd2308b authored by Matthew K Defenderfer's avatar Matthew K Defenderfer
Browse files

acheck to see if mode exists in the dataset in the first place

parent f5686380
No related branches found
No related tags found
1 merge request!9Draft: Partition parquet dataset for sync with s5cmd
......@@ -34,7 +34,8 @@ def main():
ddf = dd.read_parquet(input_parquet)
ddf = ddf.loc[ddf['path'].str.startswith(filter)].sort_values('path')
ddf = ddf.loc[~ddf['mode'].str.startswith('d')]
if 'mode' in ddf.columns:
ddf = ddf.loc[~ddf['mode'].str.startswith('d')]
ddf['cmd'] = ddf['path'].map(lambda x: create_sync_cmd(x, filter=filter, dest=dest), meta=str)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment