Skip to content

Implement 'addToStore' #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Implement 'addToStore' #53

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d51cced
Implement 'addToStore'
layus Nov 29, 2019
7054f40
Refactor according to comments
layus Nov 29, 2019
501099a
stash stalled development
layus Jan 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions hnix-store-remote/hnix-store-remote.cabal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,34 @@ library
hs-source-dirs: src
default-language: Haskell2010
ghc-options: -Wall

test-suite format-tests
ghc-options: -rtsopts -fprof-auto
type: exitcode-stdio-1.0
main-is: Driver.hs
other-modules:
Operations
hs-source-dirs:
tests
build-depends:
hnix-store-core
, hnix-store-remote
, base
, base64-bytestring
, binary
, bytestring
, containers
, directory
, mtl
, network
, process
, text
, unix
, unordered-containers
, tasty
, tasty-discover
, tasty-hspec
, tasty-hunit
, tasty-quickcheck
, temporary
default-language: Haskell2010
34 changes: 34 additions & 0 deletions hnix-store-remote/src/System/Nix/Store/Remote.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,33 @@
{-# LANGUAGE TypeApplications #-}
module System.Nix.Store.Remote (
runStore
, addToStore
, syncWithGC
, optimiseStore
, verifyStore
) where

import Control.Monad
import Control.Monad.Except

import Data.Text.Lazy as TL
import Data.Text.Lazy.Encoding as TL

import System.Nix.Hash
import System.Nix.Nar
import System.Nix.StorePath
import System.Nix.Store.Remote.Types
import System.Nix.Store.Remote.Protocol
import System.Nix.Store.Remote.Util
import System.Nix.Util

import Data.ByteString.Lazy as LBS


type RepairFlag = Bool
type CheckFlag = Bool
type RecursiveFlag = Bool
type PathFilter = FilePath -> Bool

syncWithGC :: MonadStore ()
syncWithGC = void $ simpleOp SyncWithGC
Expand All @@ -32,3 +46,23 @@ verifyStore :: CheckFlag -> RepairFlag -> MonadStore Bool
verifyStore check repair = simpleOpArgs VerifyStore $ do
putBool check
putBool repair

addToStore :: forall hashAlgo. (NamedAlgo hashAlgo, ValidAlgo hashAlgo)
=> StorePathName
-> FilePath
-> RecursiveFlag
-> PathFilter
-> RepairFlag
-> MonadStore LBS.ByteString
addToStore name srcPath recursive filter repair = do
nar <- liftIO $ localPackNar narEffectsIO srcPath -- TODO actually use filter.
runOpArgs AddToStore $ do
putByteStringLen $ strToN $ unStorePathName name
putBool $ not (recursive && algoName @hashAlgo == "sha256") -- backwards compatibility hack
Copy link
Member

@shlevy shlevy Nov 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add an algo :: HashAlgorithm field to ValidAlgo instead?

Copy link
Contributor Author

@layus layus Nov 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that, but could not make it work, because then you need to define algo for Truncated, and you need to instantiate Eq for HashAlgorithm. I failed at both of these.

Copy link
Member

@shlevy shlevy Nov 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, I think the fundamental problem is we're trying to pass the hash algo at the type level at all. Why not just take a value-level hash argument here? It's not like we construct a Digest or anything ourselves.

Copy link
Contributor Author

@layus layus Nov 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we need a NamedHash to get the pass the corresponding algoName to the daemon.
How do you type a function that takes a HashAlgorith whose corresponding type is a a NamedHash instance ? And how do you access this instance from the value.

We could just take a string, but that would be kinda defeating the purpose of all of this.

Copy link
Member

@shlevy shlevy Nov 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Argh, right. I'd say we should just have a value-level hashName function but then we'd want to exclude truncated hashes.

I'm torn between getting rid of this type level machinery altogether or separating out the base hash algorithms (sha, md5, etc.) from the digest types (base algo | truncated). Thoughts?

Copy link
Contributor Author

@layus layus Nov 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, I only ever programmed in haskell for the past "advent of code". I am therefore no expert in the habits of haskell programming.

But, if we want to mimic the remote store api as close as possible, then passing a real hash value as an argument would make sense. The current type annotation is strange.
Also, the Truncated stuff is weird to handle.
I was also surprised at how easy it is to create a "fake" hash. Turning the returned string into a correctly typed store path was very easy, but could be incorrect. A type system that can be easily escaped looks a bit like a "nullptr" in other languages.

May I propose to at least remove Truncated from the datatype ? It has no use outside of store paths while the other algos are used everywhere in nixpkgs.
That way we can make the resulting HashAlgorithms Eq.

I guess we will get a better idea of what to do after implementing several api calls.

Copy link
Member

@shlevy shlevy Dec 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I will give this separation a shot.

Copy link
Contributor Author

@layus layus Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The further I think about this, the more it seems that using algoName could make sense. What does not make sense however, is how to create a valid StorePath from the string returned by the nix daemon.

The way I do it is not valid, as it creates a Digest from the base32-encoded hash. If you print a path generated like that, it re-encodes the digest to base32, making it too long.

https://github.com/haskell-nix/hnix-store/pull/53/files#diff-002378d5e26f547b360b3e048e2d2f9fR81-R88

If you split Truncated from the hash methods, a good use case is to implement makeStorePath :: BS.ByteString -> Maybe (StorePath storeDir)

putBool recursive
putByteStringLen $ strToN (algoName @hashAlgo)
putNar nar

sockGetStr
where
strToN = TL.encodeUtf8 . TL.fromStrict
1 change: 1 addition & 0 deletions hnix-store-remote/tests/Driver.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{-# OPTIONS_GHC -F -pgmF tasty-discover #-}
47 changes: 47 additions & 0 deletions hnix-store-remote/tests/Operations.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TypeApplications #-}

module Operations where

import Control.Monad
import Control.Monad.Except
import Control.Monad.Reader
import Control.Monad.State
import Data.Maybe

import Data.Proxy
import Data.Text.Encoding ( encodeUtf8 )
import Data.Text.Lazy as TL
import Data.Text.Lazy.Encoding as TL

import System.Nix.Hash
import System.Nix.Nar
import System.Nix.StorePath
import System.Nix.Store.Remote
import System.Nix.Store.Remote.Types
import System.Nix.Store.Remote.Protocol
import System.Nix.Store.Remote.Util
import System.Nix.Util

import Test.Tasty as T
import Test.Tasty.Hspec
import qualified Test.Tasty.HUnit as HU
import Test.Tasty.QuickCheck
import Text.Read (readMaybe)

spec_addToStore :: Spec
spec_addToStore = do

describe "addToStore remote operation" $ do

it "uploads correctly" $ do
let name = fromJust $ makeStorePathName "test-recursive-add"
let srcPath = "./tests/data/add-recursive"
let recursive = True
let filter path = False -- not used anyway.
let repair = False
res <- runStore $ addToStore @'SHA256 name srcPath recursive filter repair
res `shouldBe` (Right "/nix/store/0mbh3xdb9fkqb2i3iwv6hhz7qiicca83-test-recursive-add",[Last])

Empty file added hnix-store-remote/tests/data/add-recursive/a/c
View file
Open in desktop
Empty file.
Empty file added hnix-store-remote/tests/data/add-recursive/b
View file
Open in desktop
Empty file.