WingNews logo WingNews GitHub [2]

SirOibaf's comments

SirOibaf | 4 years ago | on: Ask HN: Who is hiring? (August 2021)

Logical Clocks | Mid/Senior Frontend Developer (React) | Europe (Stockholm or Remote) | Full-time | https://logicalclocks.com

Logical Clocks is the company behind Hopsworks. Hopsworks is an open-source feature store that allows teams to organize and scale their feature engineering efforts. (https://hopsworks.ai) We recently raised our series A (5M euro) and we are expanding the teams. We are looking for a mid/senior frontend developer. Our frontend stack includes Typescript, React JS, Redux.

Your responsibilities will be to lead the development the new Hopsworks frontend, lead the technical discussions/decisions and help with the expansion of the UI/UX team.

If you want to apply/have any questions you can reach me at fabio [at] logicalclocks [dot] com or through our website: https://www.logicalclocks.com/job-positions/senior-front-end...

SirOibaf | 5 years ago | on: HopsFS: 100x Times Faster Than AWS S3

I'd say that the main difference with ObjectiveFS are metadata operations. From the documentation of ObjectiveFS:

`doesn’t fully support regular file system semantics or consistency guarantees (e.g. atomic rename of directories, mutual exclusion of open exclusive, append to file requires rewriting the whole file and no hard links).`

HopsFS does provide strongly consistent metadata operations like atomic directory rename, which is essential if you are running frameworks like Apache Spark.

SirOibaf | 5 years ago | on: HopsFS: 100x Times Faster Than AWS S3

Not really, but you can try it out on https://hopsworks.ai

It's conceptually similar to EMR in the way it works. You connect your AWS account and we'll deploy a cluster there. HopsFS will run on top a S3 bucket in your organization. You get a fully featured Spark environment (With metrics and logging included - no need for cloudwatch). UI with Jupyter notebooks, the Hopsworks feature store and ML capabilities that EMR does not provide.

SirOibaf | 5 years ago | on: How to build your own feature store for ML

You can see the Hopsworks feature store as a repository of curated features ready to be used in ML models. Or, a middle layer between data engineers and data scientists: - Data engineers write the data pipelines with the transformations and publish the features on the feature store. - Data scientists browse the feature store, pick which features they need and build the model

In the Hopsworks Feature Store we group features together in feature groups. Feature groups can be then joined to create training datasets. (You can also select a subset of features from a feature group) Training datasets are stored in a ML Framework friendly format (e.g. TFRecords if you are using TensorFlow) and you can feed them directly to your model.

If you are interested, we have a longer blog post explaining the core concepts of the Hopsworks feature store: https://www.logicalclocks.com/blog/feature-store-the-missing...

SirOibaf | 7 years ago | on: Integrating NVMe Disks in HopsFS (HDFS)

"HDFS and S3 are designed around large blocks (optimized to overcome slow random I/O on disks), while new NVMe hardware supports fast random disk I/O (and potentially small blocks sizes). "

Interesting solution presented here - keep the block size constant, but put the small files on NVMe disks.

page 1