hadoop - Retrieve only file name into a table in hive -


i need retrieve filename.txt linux path , insert filename table column in hive. possible retrieve file name path , insert hive table using virtual columns? please advice!

e.g. of path /home/usr/path/filename.txt , insert filename table. create table t( name string);

thanks!

if want run against hdfs - command - awk -f "/" '{print $nf}' file name.

[cloudera@quickstart ~]$ hadoop fs -ls /user/cloudera/departments|awk -f "/" '{print $nf}'|egrep -v 'found|_success' part-m-00000 part-m-00001 [cloudera@quickstart ~]$  

if want run against local file system - command - ls -1 give file name. can use awk -f "/" '{print $nf}'

you can create shell script as: (uncomment hive statements)

#!/bin/sh  files=`hadoop fs -ls /user/cloudera/departments|awk -f "/" '{print $nf}'|egrep -v 'found|_success'`  file in $files     #hive -e "insert table t(name) values (\"$file\");"    echo "insert table t(name) values (\"$file\");" done 

should instert in hive table:

[cloudera@quickstart ~]$ ./test.sh  insert table t(name) values ("part-m-00000"); insert table t(name) values ("part-m-00001"); 

Comments

Popular posts from this blog

jOOQ update returning clause with Oracle -

java - Warning equals/hashCode on @Data annotation lombok with inheritance -

java - BasicPathUsageException: Cannot join to attribute of basic type -