diff --git a/README.md b/README.md index 6bc86a2..37f8939 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ A simple key-value storage. -It was created for learning purposes: I wanted to know Go more and create my own small database. +It was created for learning purposes: I wanted to learn a bit of Go and create my own small database. -Not for production usage. :) +Not intended for production use. :) Implemented storage types: @@ -17,7 +17,7 @@ Implemented storage types: ## Usage -By default it uses `lsmt.Storage`, but you can change it in the [main.go](db/main.go). +By default it uses `lsmt.Storage`, but you can change this in the [main.go](db/main.go). ### Command-line interface @@ -68,36 +68,29 @@ func main() { } ``` -More about all this configuration options you can read in the `lsmt.Storage` section below. +More information about all these configuration options can be found in the `lsmt.Storage` section below. ## Internals -The database supports four different storage types. +The database supports different storage types. ### memory.Storage -It's a simple hash map which holds all in the memory. Nothing else :) +It's a simple hash map that holds everything in memory. ### file.Storage -It stores all information in the file. When you add a new entry, it just appends key and value to the file. So it's very fast to add new information. -But when you are trying to get the key, it scans the whole file (and from the beginning, not from the end) to find the latest key. So reading is slow. +It stores all information in a file. When you add a new entry, it simply appends the key and value to the file. So it's very fast to add new information. However, when you try to retrieve a key, it scans the entire file (starting from the beginning, not the end) to find the latest key. Therefore, reading is slow. ### indexedfile.Storage -It is a `FileStorage` with a simple index (hash map). When you add a new key, it saves offset in bytes to the map in the memory. To process `get` command it checks the index, finds offset in bytes and reads only a piece of the file. Writing and reading are fast, but you need a lot of memory to keep all keys in it. +This is a FileStorage with a simple index (hash map). When you add a new key, it saves the offset in bytes to the map in memory. To process the get command, it checks the index, finds the offset in bytes, and reads only a piece of the file. Writing and reading are fast, but you need a lot of memory to keep all keys in it. ### lsmt.Storage -The most interesting part of this project. :) +It stores all data in sorted string tables (SSTables), which are essentially binary files. It supports sparse indexes, so you don't need a lot of memory to store all your keys like in indexedfile.Storage. -It's something similar to Google's LevelDB or Facebook's RocksDB. -It keeps all data in sorted string tables (SSTable) which are basically binary files. -Supports sparse indexes, so you don't need a lot of memory to store all your keys like in `indexedfile.Storage`. - -But it will be slower than `indexedfile.Storage`, because it uses a red-black tree to store sparse index and it checks all SSTables when you retrieve a value because it can't say that it doesn't have this key without checking the SSTables on a disk. - -To make it faster in this situation, we can use a Bloom filter. +However, it will be slower than indexedfile.Storage because it uses a red-black tree to store a sparse index and checks all SSTables when you retrieve a value. This is because it can't determine whether it has this key without checking the SSTables on disk. It could probably use a Bloom filter for that. ```none +------------+ @@ -144,7 +137,7 @@ Main parts: * Flush queue (list of memtables) * Flusher (dumps a memtable to a disk) * SSTables storage (main storage for the data) -* Compaction (background process to remove old keys which were updated) +* Compaction (background process to remove old keys that were updated) #### GET process @@ -153,7 +146,7 @@ Main parts: 3. Check SSTables It checks all these parts in this order to be sure that it returns the latest version of the key. -Each SSTable has its own index (red-black tree). It can be sparse: it will not keep each key-offset pair in the index, +Each SSTable has its own index. It can be sparse: it will not keep each key-offset pair in the index, but it will store keys every N bytes. We can do this because SSTable files are sorted and read-only. When we need to find a key, we find its offset or closest minimal to this key. After we can load part of the file into memory and find the value for the key. @@ -164,18 +157,17 @@ key, we find its offset or closest minimal to this key. After we can load part o #### Flush -When memtable becomes bigger than some threshold, core component puts it to the flush queue and initializes a new memtable. -Flusher is a background process which checks the queue and dumps memtables as sstables to a disk. +When the memtable becomes bigger than some threshold, the core component puts it to the flush queue and initializes a new memtable. +The flusher is a background process that checks the queue and dumps memtables as SSTables to disk. #### Compaction -It's a periodical background process. -It merges small SSTable files into a bigger one and removes old key-value pairs which can be removed. +It's a periodical background process that merges small SSTable files into a larger one and removes old key-value pairs that can be removed. #### SSTables storage -It's a disk storage. On start-up time `mdb` checks this folder and registers all files and builds indexes. -Files are read-only, `mdb` never change them. It can only merge them into a big one file, but without modifying old files. +It's a disk storage. During start-up, mdb checks this folder, registers all files, and builds indexes. +Files are read-only; mdb never changes them. It can only merge them into a larger file, but without modifying old files. #### File format @@ -193,13 +185,13 @@ entry_type: ##### Configuration ```none -CompactionEnabled bool // you can disable background compaction process -MinimumFilesToCompact int // how many files does it need to start compaction process +CompactionEnabled bool // Enable/disable the background compaction process +MinimumFilesToCompact int // How many files are needed to start the compaction process MaxMemtableSize int64 // max size for memtable -MaxCompactFileSize int64 // do not compact files bigger than this size -SSTableReadBufferSize int // read buffer size: database will build indexes each - // bytes. If you want to have non-sparse index - // put 1 here +MaxCompactFileSize int64 // Do not compact files bigger than this size +SSTableReadBufferSize int // Read buffer size: the database will build indexes every + // bytes. If you want to have a non-sparse index + // put 1 here ``` #### performance test mode diff --git a/cmd/perfomance.go b/cmd/perfomance.go index 8fd58a7..ff8e889 100644 --- a/cmd/perfomance.go +++ b/cmd/perfomance.go @@ -8,26 +8,26 @@ import ( "github.com/alexander-akhmetov/mdb/pkg" ) -// now it's better to start performance test with external output filtering: +// It's better to start the performance test with external output filtering: // // go run db/*.go -p 2>&1 | grep -v 'DEBUG' // -// Otherwise it will print a lot of additional log informarion: per each inserted key. -// Later I will add log filtering to the performance test. +// Otherwise, it will print a lot of additional log information. For example, a log line for each inserted key. +// TODO: Add log filtering to the performance test. -// with this counter we will calculate how many -// inserts were made for the previous second +// With this counter, we will calculate how many +// inserts were made in the previous second. var counter = 0 -// This is an infinite loop which just writes random keys to the storage -// and every second it prints output: how many keys were inserted for the previous second, -// for example: +// This is an infinite loop that writes random keys to the storage +// and prints the output every second: how many keys were inserted. +// For example: // // 2018/08/17 07:21:39.010602 Inserted: 13141 // 2018/08/17 07:21:40.010651 Inserted: 13169 // -// It doesn't check are the inserted values valid or not. -// It just inserts key as fast as it can, nothing else. +// It doesn't check whether the inserted values are valid. +// It simply inserts keys as fast as possible, nothing more. func performanceTest(db mdb.Storage, maxKeys int, checkKeys bool) { go printStatsEverySecond() @@ -57,8 +57,8 @@ func performanceTest(db mdb.Storage, maxKeys int, checkKeys bool) { } -// printStatsEverySecond prints counter value and sets it to 0, -// then it sleeps for a second and does the same, again and again :) +// printStatsEverySecond prints the counter value, resets it, +// sleeps for a second, and repeats the process. func printStatsEverySecond() { for true == true { log.Printf("%sInserted: %v%s", colorGreen, counter, colorNeutral) diff --git a/pkg/base.go b/pkg/base.go index b75aced..da832ba 100644 --- a/pkg/base.go +++ b/pkg/base.go @@ -1,5 +1,4 @@ // Package mdb is a simple key-value database -// license that can be found in the LICENSE file. package mdb import ( diff --git a/pkg/file/file.go b/pkg/file/file.go index e772f1c..fce0da9 100644 --- a/pkg/file/file.go +++ b/pkg/file/file.go @@ -10,12 +10,12 @@ import ( var writeMutex = &sync.Mutex{} -// Storage holds all information in a file +// Storage holds all information in a file. type Storage struct { Filename string } -// Set saves given key and value +// Set saves the given key and value. func (s *Storage) Set(key string, value string) { writeMutex.Lock() defer writeMutex.Unlock() @@ -23,7 +23,7 @@ func (s *Storage) Set(key string, value string) { utils.AppendToFile(s.Filename, strToAppend) } -// Get returns a value by given key and boolean indicator that key exists +// Get returns a value for a given key and a boolean indicator of whether the key exists. func (s *Storage) Get(key string) (string, bool) { line, found := utils.FindLineByKeyInFile(s.Filename, key) if found { @@ -33,7 +33,7 @@ func (s *Storage) Get(key string) (string, bool) { return "", false } -// Start initializes Storage and creates file if needed +// Start initializes Storage and creates a file if needed. func (s *Storage) Start() { log.Println("[INFO] Starting file storage") utils.StartFileDB() diff --git a/pkg/file/file_test.go b/pkg/file/file_test.go index 1125c71..9ee8a24 100644 --- a/pkg/file/file_test.go +++ b/pkg/file/file_test.go @@ -31,7 +31,7 @@ func TestFileStorage(t *testing.T) { assert.Equal(t, expContent, content, "File content wrong") - // now let's read content from this file + // Let's read the content of this file value, exists := storage.Get(testKey2) assert.Equal(t, testValue2, value, "Wrong value") diff --git a/pkg/indexed_file/indexed_file.go b/pkg/indexed_file/indexed_file.go index f4e8028..cfe33b6 100644 --- a/pkg/indexed_file/indexed_file.go +++ b/pkg/indexed_file/indexed_file.go @@ -13,13 +13,13 @@ import ( var writeMutex = &sync.Mutex{} -// Storage holds all in a file +// Storage holds data in a file type Storage struct { Filename string index map[string]int64 } -// Set saves given key and value +// Set saves the given key and value. func (s *Storage) Set(key string, value string) { writeMutex.Lock() defer writeMutex.Unlock() @@ -30,7 +30,7 @@ func (s *Storage) Set(key string, value string) { utils.AppendToFile(s.Filename, strToAppend) } -// Get returns a value by given key and boolean indicator that key exists +// Get returns a value for a given key and a boolean indicator of whether the key exists. func (s *Storage) Get(key string) (string, bool) { var line string if offset, ok := s.index[key]; ok { @@ -42,7 +42,7 @@ func (s *Storage) Get(key string) (string, bool) { return "", false } -// Start initializes Storage, creates file if needed and rebuilds index +// Start initializes the Storage, creates the file if needed and rebuilds the index. func (s *Storage) Start() { log.Println("[INFO] Starting indexed file storage") utils.StartFileDB() @@ -52,8 +52,8 @@ func (s *Storage) Start() { log.Println("[DEBUG] Storage: started") } -// rebuildIndex reads all file and builds initial index -// it will be very slow for large files +// rebuildIndex reads the file and builds an initial index. +// It is slow for large files. func (s *Storage) rebuildIndex() { s.index = map[string]int64{} diff --git a/pkg/indexed_file/indexed_file_test.go b/pkg/indexed_file/indexed_file_test.go index d341c93..1338ffd 100644 --- a/pkg/indexed_file/indexed_file_test.go +++ b/pkg/indexed_file/indexed_file_test.go @@ -31,7 +31,7 @@ func TestIndexedFileStorage(t *testing.T) { assert.Equal(t, expContent, content, "File content wrong") - // now let's read content from this file + // Let's read the content of this file value, exists := storage.Get(testKey) assert.Equal(t, testValue, value, "Wrong value") @@ -64,12 +64,12 @@ func TestIndexedFileStorageIndexBuild(t *testing.T) { storage.Set(testKey, testValue) storage.Set(testKey2, testValue2) - // clean index and check it + // clean the index and check it storage.index = map[string]int64{} assert.Equal(t, int64(0), storage.index[testKey], "index must be empty") assert.Equal(t, int64(0), storage.index[testKey2], "index must be empty") - // now let's build index again + // build the index again storage.Stop() storage.Start() assert.Equal(t, int64(0), storage.index[testKey], "wrong index offset") diff --git a/pkg/lsmt/binfile.go b/pkg/lsmt/binfile.go index 85f9423..426b3ff 100644 --- a/pkg/lsmt/binfile.go +++ b/pkg/lsmt/binfile.go @@ -10,13 +10,12 @@ import ( const filePermissions = 0600 -// binScanner scans binary file and splits data into -// entry.DBEntry automatically +// binScanner scans a binary file and automatically splits data into entry.DBEntry. type binScanner struct { *bufio.Scanner } -// appendBinaryToFile writes key-value in binary format +// appendBinaryToFile writes key-value pairs in binary format. func appendBinaryToFile(filename string, entry *entry.DBEntry) { // todo: move to entry file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, filePermissions) diff --git a/pkg/lsmt/binfile_test.go b/pkg/lsmt/binfile_test.go index 09924fe..474e5d9 100644 --- a/pkg/lsmt/binfile_test.go +++ b/pkg/lsmt/binfile_test.go @@ -12,7 +12,7 @@ import ( func TestAppendBinaryToFile(t *testing.T) { // test appendBinaryToFile - // file must exists + // file must exist testutils.SetUp() defer testutils.Teardown() @@ -30,11 +30,11 @@ func TestAppendBinaryToFile(t *testing.T) { Value: value, }) - // check that keys are added - expKeysMap := [][2]string{[2]string{key, value}} + // check that the keys are added + expKeysMap := [][2]string{{key, value}} testutils.AssertKeysInFile(t, filename, expKeysMap) - // check binary content + // check the binary content bytes := testutils.ReadFileBinary(filename) expBytes := []byte{0x0, 0x0, 0x0, 0x0, 0x8, 0x0, 0x0, 0x0, 0xa, 0x74, 0x65, 0x73, 0x74, 0x2d, 0x6b, 0x65, 0x79, 0x74, 0x65, 0x73, 0x74, 0x2d, 0x76, 0x61, 0x6c, 0x75, 0x65} assert.Equal(t, expBytes, bytes) @@ -68,7 +68,7 @@ func TestNewBinFileScanner(t *testing.T) { readBufferSize := 1024 scanner := newBinFileScanner(f, readBufferSize) - // we have only one key-value in the file + // we have only one key-value pair in the file e, err := scanner.ReadEntry() assert.Nil(t, err) assert.Equal(t, &entry.DBEntry{Key: key, Value: value}, e) diff --git a/pkg/lsmt/compaction.go b/pkg/lsmt/compaction.go index fb639cc..6ff99cc 100644 --- a/pkg/lsmt/compaction.go +++ b/pkg/lsmt/compaction.go @@ -13,9 +13,9 @@ var compactionMutex = &sync.Mutex{} const ssTableReadBufferSize = 4096 -// compact finds N SSTables in the workDir, -// which are can be merged together (they are must be smaller some limit) -// and merges them into a one bigger SSTable, then it removes old files +// compact finds N SSTables in the workDir +// that can be merged together (they must be smaller than some limit) +// and merges them into one bigger SSTable. Then it removes the old files. func compact(workDir string, tmpDir string, minimumFilesToCompact int, maxCompactFileSize int64) (string, string, string, bool) { compactionMutex.Lock() defer compactionMutex.Unlock() @@ -35,7 +35,7 @@ func compact(workDir string, tmpDir string, minimumFilesToCompact int, maxCompac return fFile, sFile, tmpFilePath, true } -// merge merges files into a one +// merge merges files into one. func merge(fFile string, sFile string, mergeTo string) { log.Printf("[DEBUG] Merging %s + %s => %s", fFile, sFile, mergeTo) @@ -58,7 +58,7 @@ func merge(fFile string, sFile string, mergeTo string) { sEntry, _ := secondScanner.ReadEntry() for true == true { - // compare files line by line and add to the new file only last keys + // Compare files line by line and add only the latest keys to the new file. for (sEntry.Key > fEntry.Key && fEntry.Key != "") || (fEntry.Key != "" && sEntry.Key == "") { appendBinaryToFile(mergeTo, fEntry) fEntry, _ = firstScanner.ReadEntry() @@ -67,8 +67,8 @@ func merge(fFile string, sFile string, mergeTo string) { for (sEntry.Key <= fEntry.Key && sEntry.Key != "") || (fEntry.Key == "" && sEntry.Key != "") { appendBinaryToFile(mergeTo, sEntry) for sEntry.Key == fEntry.Key { - // if keys are equal, we need to read next first key too, - // otherwise we will save it again in this loop + // If keys are equal, we need to read the next first key too, + // otherwise we will save it again in this loop. fEntry, _ = firstScanner.ReadEntry() } sEntry, _ = secondScanner.ReadEntry() @@ -79,8 +79,8 @@ func merge(fFile string, sFile string, mergeTo string) { } } -// getTwoFilesToCompact returns paths to to files which we can merge -// and boolean third argument which indicates can we merge files or not +// getTwoFilesToCompact returns paths to two files that we can merge +// and a boolean indicating whether we can merge the files or not. func getTwoFilesToCompact(dir string, minimumFilesToCompact int, maxFileSize int64) (string, string, bool) { allFiles := listSSTables(dir) diff --git a/pkg/lsmt/compaction_test.go b/pkg/lsmt/compaction_test.go index be06d18..2051729 100644 --- a/pkg/lsmt/compaction_test.go +++ b/pkg/lsmt/compaction_test.go @@ -10,7 +10,7 @@ import ( func TestCompactionWithoutFiles(t *testing.T) { // test compact() with only one file - // it should do nothing + // it should not do anything testutils.SetUp() defer testutils.Teardown() @@ -39,14 +39,14 @@ func TestSimpleCompaction(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{"k1", "v1"}, - [2]string{"k2", "v2"}, + {"k1", "v1"}, + {"k2", "v2"}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1.sstable", [][2]string{ - [2]string{"k1", "v11"}, + {"k1", "v11"}, }, ) testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/2.sstable", [][2]string{}) @@ -64,30 +64,30 @@ func TestSimpleCompaction(t *testing.T) { assert.Equal(t, ".test/lsmt_data/sstables/tmp/1.sstable", c) expData := [][2]string{ - [2]string{"k1", "v11"}, - [2]string{"k2", "v2"}, + {"k1", "v11"}, + {"k2", "v2"}, } testutils.AssertKeysInFile(t, ".test/lsmt_data/sstables/tmp/1.sstable", expData) } func TestSimpleCompactionWithSameKeys(t *testing.T) { // test compaction process with same keys in different files: - // it should save only latest keys + // it should save only the latest key-value pairs. testutils.SetUp() defer testutils.Teardown() testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{"k1", "01"}, - [2]string{"k2", "02"}, + {"k1", "01"}, + {"k2", "02"}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1.sstable", [][2]string{ - [2]string{"k1", "11"}, - [2]string{"k2", "22"}, + {"k1", "11"}, + {"k2", "22"}, }, ) @@ -104,8 +104,8 @@ func TestSimpleCompactionWithSameKeys(t *testing.T) { assert.Equal(t, ".test/lsmt_data/sstables/tmp/1.sstable", c) expData := [][2]string{ - [2]string{"k1", "11"}, - [2]string{"k2", "22"}, + {"k1", "11"}, + {"k2", "22"}, } testutils.AssertKeysInFile(t, ".test/lsmt_data/sstables/tmp/1.sstable", expData) } @@ -119,19 +119,19 @@ func TestComplexCompaction(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{"k1", "1"}, - [2]string{"k2", "2"}, - [2]string{"k3", "3"}, - [2]string{"k3", "33"}, + {"k1", "1"}, + {"k2", "2"}, + {"k3", "3"}, + {"k3", "33"}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1.sstable", [][2]string{ - [2]string{"k1", "11"}, - [2]string{"k3", "333"}, - [2]string{"k5", "5"}, - [2]string{"k6", "6"}, + {"k1", "11"}, + {"k3", "333"}, + {"k5", "5"}, + {"k6", "6"}, }, ) testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/2.sstable", [][2]string{}) @@ -139,11 +139,11 @@ func TestComplexCompaction(t *testing.T) { compact(".test/lsmt_data/sstables/", ".test/lsmt_data/sstables/tmp/", 2, defaultMaxCompactFileSize) expData := [][2]string{ - [2]string{"k1", "11"}, - [2]string{"k2", "2"}, - [2]string{"k3", "333"}, - [2]string{"k5", "5"}, - [2]string{"k6", "6"}, + {"k1", "11"}, + {"k2", "2"}, + {"k3", "333"}, + {"k5", "5"}, + {"k6", "6"}, } testutils.AssertKeysInFile(t, ".test/lsmt_data/sstables/tmp/1.sstable", expData) } @@ -157,10 +157,10 @@ func TestCompactionWithOneEmptyFile(t *testing.T) { assert.True(t, testutils.IsFileExists(".test/lsmt_data/sstables/0.sstable")) secondFileKeys := [][2]string{ - [2]string{"k1", "11"}, - [2]string{"k3", "333"}, - [2]string{"k5", "5"}, - [2]string{"k6", "6"}, + {"k1", "11"}, + {"k3", "333"}, + {"k5", "5"}, + {"k6", "6"}, } testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/1.sstable", secondFileKeys) testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/2.sstable", [][2]string{}) @@ -178,8 +178,8 @@ func TestCompactionWithEmptySecondFile(t *testing.T) { defer testutils.Teardown() firstFileKeys := [][2]string{ - [2]string{"k1", "1"}, - [2]string{"k3", "3"}, + {"k1", "1"}, + {"k3", "3"}, } testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/0.sstable", firstFileKeys) testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/1.sstable", [][2]string{}) diff --git a/pkg/lsmt/flush.go b/pkg/lsmt/flush.go index c988179..c048dc8 100644 --- a/pkg/lsmt/flush.go +++ b/pkg/lsmt/flush.go @@ -9,15 +9,15 @@ import ( "github.com/alexander-akhmetov/mdb/pkg/utils" ) -// flusher is a struct which holds information about -// memtable which we flush to a disk +// flusher is a struct that holds information about +// the memtable we flush to disk. type flusher struct { sstablesDir string memtable *memtable } -// flush dumps data from flusher.memtable to a new SSTable on a disk -// SSTable's name defined as a "{flusher.timestamp}.sstable" +// flush dumps data from flusher.memtable to a new SSTable on disk. +// The SSTable's name is defined as "{flusher.timestamp}.sstable". func (f *flusher) flush() string { log.Printf("[DEBUG] Starting memtable flushing process for aolog=%s", f.memtable.logFilename) file, err := os.OpenFile(f.filename(), os.O_APPEND|os.O_WRONLY, filePermissions) @@ -46,8 +46,8 @@ func (f *flusher) flush() string { return file.Name() } -// filename returns full path to an SSTable file -// in which flusher writes memtable's data +// filename returns the full path to an SSTable file +// where flusher writes the memtable's data. func (f *flusher) filename() string { return filepath.Join( f.sstablesDir, diff --git a/pkg/lsmt/internal/rbt/rbt.go b/pkg/lsmt/internal/rbt/rbt.go index 8c9b695..bf0a279 100644 --- a/pkg/lsmt/internal/rbt/rbt.go +++ b/pkg/lsmt/internal/rbt/rbt.go @@ -14,7 +14,7 @@ func NewRBTree() *RedBlackTree { return &RedBlackTree{rbt.NewWithStringComparator()} } -// GetClosest returns value of by key or the closest minimal one +// GetClosest returns the value of the key or the closest minimal one. func (tree *RedBlackTree) GetClosest(value string) int { return tree.getClosestValue(tree.Root, value, -1) } diff --git a/pkg/lsmt/internal/rbt/rbt_test.go b/pkg/lsmt/internal/rbt/rbt_test.go index 70e94ba..be21361 100644 --- a/pkg/lsmt/internal/rbt/rbt_test.go +++ b/pkg/lsmt/internal/rbt/rbt_test.go @@ -7,8 +7,8 @@ import ( ) func TestFindClosestValueInRBTree(t *testing.T) { - // GetClosest() should return value by given key - // or by closest key (min) + // GetClosest() should return the value for the given key + // or for the closest key (min). tree := NewRBTree() tree.Put("key_a", 0) diff --git a/pkg/lsmt/lsmt.go b/pkg/lsmt/lsmt.go index 5bb49a0..bfa6370 100644 --- a/pkg/lsmt/lsmt.go +++ b/pkg/lsmt/lsmt.go @@ -16,13 +16,13 @@ import ( const defaultMaxMemtableSize int64 = 256 const defaultMaxCompactFileSize int64 = 1024 * 1024 * 10 -// locks access to change the memtablesFlushQueue +// Prevents changing the memtablesFlushQueue var flushMutex = &sync.Mutex{} -// locks access to change the ssTables list +// Prevents changing the ssTables list var ssTablesListMutex = &sync.Mutex{} -// locks access to search in the ssTables list +// Locks access to the ssTables list var ssTablesAccessMutex = &sync.Mutex{} // StorageConfig holds all configuration of the storage @@ -42,7 +42,7 @@ type StorageConfig struct { tmpDir string } -// Storage holds all in a file +// Storage holds data in ss tables type Storage struct { Config StorageConfig @@ -52,13 +52,13 @@ type Storage struct { memtablesFlushQueue []*memtable } -// Set saves given key and value +// Set saves the given key and value. func (s *Storage) Set(key string, value string) { s.flushmemtableIfNeeded() s.memtable.Set(key, value) } -// flushmemtableIfNeeded checks if memtable is bigger than limit size and puts it to the flush queue if yes +// flushmemtableIfNeeded checks if the memtable is bigger than the limit size and puts it into the flush queue if yes. func (s *Storage) flushmemtableIfNeeded() { if s.memtable.Size() > s.Config.MaxMemtableSize { log.Println("[DEBUG] memtable is too big: putting it to flush queue") @@ -79,9 +79,9 @@ func (s *Storage) flushmemtableIfNeeded() { } } -// appendToFlushQueue inserts wmemtable to a memtablesFlushQueue to the first place (prepend) -// we need to keep memtablesFlushQueue ordered by memtable age (we keep descending order: newest first) -// so we will check memtables from the beginning if we want to find some key +// appendToFlushQueue inserts wmemtable into the memtablesFlushQueue at the first place (prepend). +// We need to keep the memtablesFlushQueue ordered by memtable age (descending order: newest first), +// so we will check memtables from the beginning if we want to find some key. func (s *Storage) appendToFlushQueue(m *memtable) { // we must lock this mutex to obtain exclusive access to the flush queue flushMutex.Lock() @@ -92,7 +92,7 @@ func (s *Storage) appendToFlushQueue(m *memtable) { s.memtablesFlushQueue = append([]*memtable{m}, s.memtablesFlushQueue...) } -// Get returns a value by given key and boolean indicator that key exists +// Get returns a value for the given key and a boolean indicator of whether the key exists. func (s *Storage) Get(key string) (value string, exists bool) { value, exists = s.memtable.Get(key) @@ -109,7 +109,7 @@ func (s *Storage) Get(key string) (value string, exists bool) { return value, exists } -// getFromFlushQueue tries to find given key in the flush queue memtables +// getFromFlushQueue tries to find the given key in the flush queue memtables. func (s *Storage) getFromFlushQueue(key string) (string, bool) { value := "" found := false @@ -127,8 +127,8 @@ func (s *Storage) getFromFlushQueue(key string) (string, bool) { return value, found } -// getFromSSTables tries to find given key in the SSTables -// it searches keys in parallel in all SSTables +// getFromSSTables tries to find the given key in the SSTables. +// It searches for keys in parallel in all SSTables. func (s *Storage) getFromSSTables(key string) (string, bool) { value := "" found := false @@ -217,8 +217,8 @@ func (s *Storage) Start() { log.Println("[INFO] Storage ready") } -// restoreFlushQueue reads flush queue directory and restores memtables -// from files (aolog) in this directory to the memtablesFlushQueue +// restoreFlushQueue reads the flush queue directory and restores memtables +// from files (aolog) in this directory to the memtablesFlushQueue. func (s *Storage) restoreFlushQueue() { log.Println("[DEBUG] Restoring flush queue...") files := utils.ListFilesOrdered(s.Config.memtablesFlushTmpDir, "") @@ -237,12 +237,12 @@ func (s *Storage) restoreFlushQueue() { log.Println("[DEBUG] Flush queue has been restored with size=", len(s.memtablesFlushQueue)) } -// initNewMemtable initializes a new memtable for the storage +// initNewMemtable initializes a new memtable for the storage. func (s *Storage) initNewMemtable() { s.memtable = newMemtable(s.Config.aoLogPath) } -// createWorkDirs creates necessary directories +// createWorkDirs creates the necessary directories. func (s *Storage) createWorkDirs() { dirs := []string{s.Config.ssTablesDir, s.Config.memtablesFlushTmpDir, s.Config.tmpDir} for _, dir := range dirs { @@ -251,7 +251,7 @@ func (s *Storage) createWorkDirs() { } } -// restoreSSTables reads the directory with SSTables and restores them to the `ssTables` attribute +// restoreSSTables reads the directory with SSTables and restores them to the `ssTables` attribute. func (s *Storage) restoreSSTables() { type result struct { @@ -269,8 +269,8 @@ func (s *Storage) restoreSSTables() { // initialize ssTables in parallel for i, file := range tablesToRestore { - // files are already ordered by name in descending order, - // later we will put this file to the end of the list + // Files are already ordered by name in descending order. + // Later, we will put this file at the end of the list. go func(position int, filename string) { defer wg.Done() s.ssTables[position] = newSSTable( @@ -287,24 +287,24 @@ func (s *Storage) restoreSSTables() { log.Println("[DEBUG] initialized sstables:", len(s.ssTables)) } -// startFlusherProcess starts flusher process which checks -// if we need to flush some memtable and flushes if needed +// startFlusherProcess starts the flusher process, which checks +// if we need to flush some memtable and flushes it if needed. func (s *Storage) startFlusherProcess() { log.Println("[DEBUG] Started flusher process") for s.running == true { - // lock mutex, so there will be no new memtables added - // while we are dumping memtables to disk, - // and we will be able to flush all queue and clean it + // Lock the mutex so that no new memtables are added + // while we are dumping memtables to disk. + // This ensures that we can flush the entire queue and clean it. flushMutex.Lock() - // FIFO: we iterate in reverse order to dump oldest memtables to disk first - // it allows us to serve read requests correctly: we search in the main memtable first, - // then in the "memtables to flush" queue from top to bottom (newest first) - // then in sstables + // FIFO: We iterate in reverse order to dump the oldest memtables to disk first. + // This allows us to serve read requests correctly: we search in the main memtable first, + // then in the "memtables to flush" queue from top to bottom (newest first), + // and finally in SSTables. for i := len(s.memtablesFlushQueue) - 1; i >= 0; i-- { f := newFlusher(s.memtablesFlushQueue[i], s.Config.ssTablesDir) filename := f.flush() - // it is the newest sstable, so put it to the beginning of the list + // It is the newest SSTable, so put it at the beginning of the list. ssTablesListMutex.Lock() newt := newSSTable( &ssTableConfig{ @@ -316,11 +316,11 @@ func (s *Storage) startFlusherProcess() { ssTablesListMutex.Unlock() } - // clean flush queue, since we flushed all memtables and - // mutex don't allow other goroutines to add new items to this queue + // Clean the flush queue since we flushed all memtables and + // the mutex prevents other goroutines from adding new items to this queue. s.memtablesFlushQueue = []*memtable{} - // unlock mutex and sleep some time + // Unlock the mutex and sleep for some time. flushMutex.Unlock() time.Sleep(time.Millisecond * 100) } @@ -329,15 +329,15 @@ func (s *Storage) startFlusherProcess() { func (s *Storage) startCompactionProcess() { log.Println("[DEBUG] Started compaction process") - // we need to merge two files together and place a result file to the temporary dir - // then we lock ssTables to be sure that we have exclusive access to change it, - // and move the result file to the place of the second merged one. - // We need to do this because the second file is newer - // and even if something will be broken after - we won't lose data. + // We need to merge two files together and place the result file in the temporary directory. + // Then we lock ssTables to ensure exclusive access to change it, + // and move the result file to the location of the second merged one. + // We do this because the second file is newer, + // and even if something goes wrong, we won't lose data. // - // After moving the result file, we can remove the first merged file - we don't need it anymore - // And after we can remove it's ssTable instance from the list. - // But we already don't use it automatically, since all newer keys are in the newer file + // After moving the result file, we can remove the first merged file as we don't need it anymore. + // Then we remove its ssTable instance from the list. + // However, we already don't use it automatically since all newer keys are in the newer file. for s.running == true { firstMerged, secondMerged, resultFile, isMerged := compact( s.Config.ssTablesDir, @@ -364,7 +364,7 @@ func (s *Storage) startCompactionProcess() { defer file.Close() ssTablesAccessMutex.Lock() - // move the result file to place of the second merged + // Move the result file to the location of the second merged file. err := os.Rename(resultFile, secondMerged) if err != nil { log.Printf("[ERROR] Can't move merged file from '%s' to '%s': %v", resultFile, secondMerged, err) @@ -390,14 +390,14 @@ func (s *Storage) startCompactionProcess() { ssTablesListMutex.Unlock() log.Println("[DEBUG] Compaction completed") } else { - // if we didn't merge files, let's sleep, - // but if we just merged files - we want to check, maybe we need to merge them again + // If we didn't merge files, let's sleep. + // But if we just merged files, we want to check if we need to merge them again. time.Sleep(time.Millisecond * 100) } } } -// findSSTableIndex returns index of an ssTable in the ssTables list +// findSSTableIndex returns the index of an SSTable in the ssTables list. func (s *Storage) findSSTableIndex(filename string) int { index := -1 diff --git a/pkg/lsmt/lsmt_test.go b/pkg/lsmt/lsmt_test.go index e19b8af..d8571b1 100644 --- a/pkg/lsmt/lsmt_test.go +++ b/pkg/lsmt/lsmt_test.go @@ -31,7 +31,7 @@ func TestStorage(t *testing.T) { storage.Set(testKey, testValue) expData := [][2]string{ - [2]string{testKey, testValue}, + {testKey, testValue}, } testutils.AssertKeysInFile(t, storage.memtable.logFilename, expData) @@ -56,8 +56,8 @@ func TestStorageSSTable(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, }, ) @@ -79,7 +79,7 @@ func TestStorageSSTable(t *testing.T) { } func TestStorageSSTablesOrdering(t *testing.T) { - // we will create many sstables and check that they will be used correctly + // we will create many sstables and check that they are used correctly testutils.SetUp() defer testutils.Teardown() @@ -90,20 +90,20 @@ func TestStorageSSTablesOrdering(t *testing.T) { value2 := "v2" oldValue2 := "1" - // let's create two sstables and check that we use them in correct order + // let's create two sstables and check that we use them in the correct order testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{key1, oldValue1}, - [2]string{key2, oldValue2}, + {key1, oldValue1}, + {key2, oldValue2}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1.sstable", [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, }, ) @@ -138,16 +138,16 @@ func TestStorageAOLogRestoring(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/log.aolog", [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1544288836377002.sstable", [][2]string{ - [2]string{"k1", "0"}, - [2]string{"k2", "0"}, + {"k1", "0"}, + {"k2", "0"}, }, ) @@ -182,15 +182,15 @@ func TestStorageMemtablesToFlush(t *testing.T) { // let's create aolog_tf, it should be the latest version of our keys data := [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, } testutils.CreateFileWithKeyValues(".test/lsmt_data/aolog_tf/1.aolog", data) // sstable file will have older values data = [][2]string{ - [2]string{key1, oldValue1}, - [2]string{key2, oldValue2}, + {key1, oldValue1}, + {key2, oldValue2}, } testutils.CreateFileWithKeyValues(".test/lsmt_data/sstables/0.sstable", data) @@ -244,14 +244,14 @@ func TestStorageMemtableMustBeMoreImportantThanMemtablesToFlush(t *testing.T) { // let's create aolog_tf, it should be the latest version of our keys data := [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, } testutils.CreateFileWithKeyValues(".test/lsmt_data/log.aolog", data) data = [][2]string{ - [2]string{key1, oldValue1}, - [2]string{key2, oldValue2}, + {key1, oldValue1}, + {key2, oldValue2}, } testutils.CreateFileWithKeyValues(".test/lsmt_data/aolog_tf/0.aolog", data) @@ -288,8 +288,8 @@ func TestStorageMemtablesToFlushOnly(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/aolog_tf/0.aolog", [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, }, ) @@ -430,16 +430,16 @@ func TestStorageCompaction(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{key1, oldValue1}, - [2]string{key2, oldValue2}, + {key1, oldValue1}, + {key2, oldValue2}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1.sstable", [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, }, ) @@ -489,24 +489,24 @@ func TestStorageCompactionForOldFiles(t *testing.T) { testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/0.sstable", [][2]string{ - [2]string{key1, oldestValue1}, - [2]string{key2, oldestValue2}, + {key1, oldestValue1}, + {key2, oldestValue2}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/1.sstable", [][2]string{ - [2]string{key1, oldValue1}, - [2]string{key2, oldValue2}, + {key1, oldValue1}, + {key2, oldValue2}, }, ) testutils.CreateFileWithKeyValues( ".test/lsmt_data/sstables/2.sstable", [][2]string{ - [2]string{key1, value1}, - [2]string{key2, value2}, + {key1, value1}, + {key2, value2}, }, ) @@ -536,8 +536,8 @@ func TestStorageCompactionForOldFiles(t *testing.T) { assert.False(t, testutils.IsFileExists(".test/lsmt_data/sstables/0.sstable")) expData := [][2]string{ - [2]string{key1, oldValue1}, - [2]string{key2, oldValue2}, + {key1, oldValue1}, + {key2, oldValue2}, } testutils.AssertKeysInFile(t, expectedNewSSTablePath, expData) } diff --git a/pkg/lsmt/memtable.go b/pkg/lsmt/memtable.go index e33c575..1ad37a7 100644 --- a/pkg/lsmt/memtable.go +++ b/pkg/lsmt/memtable.go @@ -16,12 +16,12 @@ var writeMutex = &sync.Mutex{} const aoLogReadBufferSize = 4096 type memtable struct { - data map[string]string // in-memory data structure to keep info before saving to disk as SSTable - logFilename string // AOLog: append-only log to restore information in case of crash - timestamp int64 //used for flush process + data map[string]string // In-memory data structure to keep information before saving to disk as SSTable. + logFilename string // AOLog: append-only log to restore information in case of a crash. + timestamp int64 // Used for the flush process. } -// set writes informartion to AOLog +// Set writes information to AOLog. func (m *memtable) Set(key string, value string) { m.appendToLog(key, value) m.data[key] = value @@ -40,7 +40,7 @@ func (m *memtable) appendToLog(key string, value string) { }) } -// get returns value of a key from memtable +// Get returns the value of a key from the memtable. func (m *memtable) Get(key string) (string, bool) { if value, ok := m.data[key]; ok { return value, true @@ -49,14 +49,14 @@ func (m *memtable) Get(key string) (string, bool) { return "", false } -// getSize returns size of a memtable in bytes -// it's needed to decide if we need to dump this memtable to a disk as SSTable or not +// Size returns the size of a memtable in bytes. +// It's needed to decide if we need to dump this memtable to disk as an SSTable or not. func (m *memtable) Size() int64 { return int64(len(m.data)) } -// restoreFromLog reads AOLog file and restores all information back to the memtable -// we use it in case of crash or when server was stopped with some informartion in the memtable +// restoreFromLog reads the AOLog file and restores all information back to the memtable. +// We use it in case of a crash or when the server was stopped with some information in the memtable. func (m *memtable) restoreFromLog() { file, err := os.OpenFile(m.logFilename, os.O_RDONLY, 0600) if os.IsNotExist(err) { @@ -108,7 +108,7 @@ func (m *memtable) Write(wr io.Writer) (n int, err error) { return n, err } -// newMemtable returns new instance of a writer +// newMemtable returns a new instance of a writer. func newMemtable(aoLogFileName string) *memtable { m := &memtable{ data: map[string]string{}, diff --git a/pkg/lsmt/memtable_test.go b/pkg/lsmt/memtable_test.go index 58f7688..4852af4 100644 --- a/pkg/lsmt/memtable_test.go +++ b/pkg/lsmt/memtable_test.go @@ -11,7 +11,7 @@ import ( func TestMemtableFlush(t *testing.T) { // test that when we flush memtable to disk it writes it correctly - // and keys are sorted + // and the keys are sorted testutils.SetUp() defer testutils.Teardown() @@ -54,23 +54,23 @@ func TestSize(t *testing.T) { m := newMemtable(".test/log") - // at first size is zero + // at first the size is zero assert.Equal(t, int64(0), m.Size()) m.Set("k1", "v1") assert.Equal(t, int64(1), m.Size()) - // let's add the same key, size must be the same + // add the same key, the size must be the same m.Set("k1", "v1") assert.Equal(t, int64(1), m.Size()) - // new key: size must be changed + // a new key: the size must change m.Set("k2", "v2") assert.Equal(t, int64(2), m.Size()) } func TestAppendOnlyLog(t *testing.T) { - // test that memtable will save correct data to append only log + // test that memtable saves correct data to append only log testutils.SetUp() defer testutils.Teardown() @@ -88,7 +88,7 @@ func TestAppendOnlyLog(t *testing.T) { data = testutils.ReadFileBinary(f) assert.Equal(t, expData, data) - // now let's dump this data: it must be the same + // dump this data: it must be the same df := ".test/dump" file, _ := os.OpenFile(df, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0600) _, err := m.Write(file) @@ -104,7 +104,7 @@ func TestAppendOnlyLog(t *testing.T) { data = testutils.ReadFileBinary(f) assert.Equal(t, expData, data) - // now let's dump this data again: + // dump this data again: // it must save only the last value for the key os.Remove(df) file, _ = os.OpenFile(df, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0600) diff --git a/pkg/lsmt/sstable.go b/pkg/lsmt/sstable.go index dbce309..2ae1098 100644 --- a/pkg/lsmt/sstable.go +++ b/pkg/lsmt/sstable.go @@ -22,7 +22,7 @@ type ssTable struct { config *ssTableConfig } -// listSSTables returns filenames ordered by last modified time (descending) +// listSSTables returns filenames ordered by last modified time in descending order. func listSSTables(dir string) []utils.FileInfo { return utils.ListFilesOrdered(dir, ".sstable") } @@ -55,7 +55,7 @@ func (s *ssTable) Get(key string) (string, bool) { return "", false } -// rebuildSparseIndex reads all file and builds initial index +// rebuildSparseIndex reads the entire file and builds the initial index. func (s *ssTable) rebuildSparseIndex() { s.index = rbt.NewRBTree() @@ -82,7 +82,7 @@ func (s *ssTable) rebuildSparseIndex() { } } -// newSSTable returns an SSTable instance which can be used to get information from this table +// newSSTable returns an SSTable instance that can be used to retrieve information from this table. func newSSTable(config *ssTableConfig) *ssTable { log.Println("[DEBUG] Initializing a new SSTable instance...") if config.readBufferSize == 0 { diff --git a/pkg/lsmt/sstable_test.go b/pkg/lsmt/sstable_test.go index b274baf..4105646 100644 --- a/pkg/lsmt/sstable_test.go +++ b/pkg/lsmt/sstable_test.go @@ -15,7 +15,7 @@ import ( ) func TestListSSTables(t *testing.T) { - // test that listSSTables returns list of files with path + // test that listSSTables returns the list of paths to the files with ss tables // and they are ordered by modification time testutils.SetUp() defer testutils.Teardown() @@ -31,13 +31,12 @@ func TestListSSTables(t *testing.T) { os.OpenFile(filepath.Join(sstablesDir, f), os.O_RDONLY|os.O_CREATE, 0600) } - // expFiles with ordered by last modified time and with full path expFiles := []utils.FileInfo{ - utils.FileInfo{ + { Name: ".test/sstables-test/another.sstable", Size: 0, }, - utils.FileInfo{ + { Name: ".test/sstables-test/file.sstable", Size: 0, }, @@ -46,7 +45,8 @@ func TestListSSTables(t *testing.T) { } func TestSSTableGet(t *testing.T) { - // test that SSTable reads content from file + // test that SSTable reads content from the file + // create two key-value pairs manually and read them with .Get method testutils.SetUp() defer testutils.Teardown() diff --git a/pkg/memory/memory.go b/pkg/memory/memory.go index b5074f7..7e09bbc 100644 --- a/pkg/memory/memory.go +++ b/pkg/memory/memory.go @@ -2,17 +2,17 @@ package memory import "log" -// Storage holds all in memory +// Storage holds data in memory type Storage struct { storage map[string]string } -// Set saves given key and value +// Set saves the given key and value. func (s *Storage) Set(key string, value string) { s.storage[key] = value } -// Get returns a value by given key +// Get returns a value for the given key. func (s *Storage) Get(key string) (string, bool) { if value, exists := s.storage[key]; exists { return value, true @@ -21,12 +21,7 @@ func (s *Storage) Get(key string) (string, bool) { return "", false } -// Delete removes key from storage -// func (s *Storage) Delete(key string) { -// delete(s.storage, key) -// } - -// Start initializes memory storage +// Start initializes the memory storage func (s *Storage) Start() { log.Println("[INFO] Starting memory storage") s.storage = map[string]string{} diff --git a/pkg/memory/memory_test.go b/pkg/memory/memory_test.go index 541f37e..638c1fc 100644 --- a/pkg/memory/memory_test.go +++ b/pkg/memory/memory_test.go @@ -48,25 +48,3 @@ func TestMemoryStorageSetGet(t *testing.T) { assert.Equal(t, testValue, value, "Wrong value") assert.True(t, exists) } - -// func TestMemoryStorageDelete(t *testing.T) { -// storage := MemoryStorage{ -// storage: map[string]string{}, -// } - -// testKey := "test-key" -// testValue := "test-value" - -// storage.Set(testKey, testValue) - -// value, exists := storage.Get(testKey) -// assert.Equal(t, testValue, value, "Wrong value") -// assert.True(t, exists) - -// storage.Delete(testKey) -// value, exists = storage.Get(testKey) -// assert.Equal(t, "", value, "Wrong value") -// assert.False(t, exists) - -// assert.Equal(t, 0, len(storage.storage), "Storage must be empty") -// } diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go index d166507..f7aac80 100644 --- a/pkg/utils/utils.go +++ b/pkg/utils/utils.go @@ -16,7 +16,7 @@ import ( const filePermissions = 0600 const pidFileName = "mdb.pid" -// GetKeyValueFromString returns key and value from string +// GetKeyValueFromString returns the key and value from a string. func GetKeyValueFromString(line string) (string, string) { splitted := strings.SplitN(line, ";", 2) if len(splitted) != 2 { @@ -25,14 +25,14 @@ func GetKeyValueFromString(line string) (string, string) { return splitted[0], strings.TrimRight(splitted[1], "\n") } -// TrimKey removes key and ";" prefix from line -// to get value only +// TrimKey removes the key and ";" prefix from the line +// to obtain only the value. func TrimKey(key string, line string) string { return strings.TrimPrefix(line, fmt.Sprintf("%s;", key)) } -// FindLineByKeyInFile returns the last line which starts with given key and boolean indicator that line has been found -// if it's false - line has not been found +// FindLineByKeyInFile returns the last line that starts with the given key and a boolean indicator of whether the line has been found. +// If false, the line has not been found. func FindLineByKeyInFile(filename string, key string) (string, bool) { file, err := os.Open(filename) if err != nil { @@ -60,7 +60,7 @@ func FindLineByKeyInFile(filename string, key string) (string, bool) { return resultLine, found } -// AppendToFile appends given string to a file with given filename +// AppendToFile appends the given string to a file with the specified filename. func AppendToFile(filename string, appendString string) { file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, filePermissions) if err != nil { @@ -74,13 +74,13 @@ func AppendToFile(filename string, appendString string) { } } -// RecreateFile removes old file and creates a new one +// RecreateFile removes the old file and creates a new one. func RecreateFile(filename string) { os.Remove(filename) CreateFileIfNotExists(filename) } -// CreateFileIfNotExists creates file and all dirs if it doesn't exist +// CreateFileIfNotExists creates the file and all necessary directories if it doesn't exist. func CreateFileIfNotExists(filename string) { dir, _ := filepath.Split(filename) CreateDir(dir) @@ -90,14 +90,14 @@ func CreateFileIfNotExists(filename string) { } } -// CreateDir creates dir like `mkdir -p` +// CreateDir creates a directory similarly to `mkdir -p`. func CreateDir(dir string) { if _, err := os.Stat(dir); os.IsNotExist(err) { os.MkdirAll(dir, os.ModePerm) } } -// GetFileSize returns file size +// GetFileSize returns the file size. func GetFileSize(filename string) int64 { file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, filePermissions) if err != nil { @@ -111,7 +111,7 @@ func GetFileSize(filename string) int64 { return fi.Size() } -// ReadLineByOffset reads line from file by given offset +// ReadLineByOffset reads a line from the file at the given offset. func ReadLineByOffset(filename string, offset int64) string { file, err := os.Open(filename) if err != nil { @@ -128,15 +128,15 @@ func ReadLineByOffset(filename string, offset int64) string { return scanner.Text() } -// StartFileDB creates temporary .pid file to lock file usage +// StartFileDB creates a temporary .pid file to lock file usage. func StartFileDB() { CheckAndCreatePIDFile(pidFileName) AppendToFile(pidFileName, fmt.Sprintf("%v", os.Getpid())) } -// CheckAndCreatePIDFile checks and creates .pid file if it does not exist -// if it exists, it will panic, because if you use it only one -// instance of the DB must be started at the same time +// CheckAndCreatePIDFile checks for the presence of a .pid file and creates it if it does not exist. +// If it exists, the function will panic, because only one +// instance of the DB should be running at the same time. func CheckAndCreatePIDFile(path string) { if _, err := os.Stat(path); err == nil { log.Panicf("Can't start the database: %s file already exists!", path) @@ -144,12 +144,12 @@ func CheckAndCreatePIDFile(path string) { CreateFileIfNotExists(path) } -// StopFileDB removes temporary .pid file +// StopFileDB removes the temporary .pid file. func StopFileDB() { RemovePIDFile(pidFileName) } -// RemovePIDFile removes .pid file +// RemovePIDFile removes the .pid file. func RemovePIDFile(path string) { err := os.Remove(path) if err != nil { @@ -167,8 +167,8 @@ type FileInfo struct { Size int64 } -// ListFilesOrdered returns filenames ordered by their name (descending) -// files *MUST* be with integer names +// ListFilesOrdered returns filenames ordered by their name in descending order. +// Files must have integer names. func ListFilesOrdered(dir string, filterBySuffix string) []FileInfo { files, err := ioutil.ReadDir(dir) if err != nil { diff --git a/pkg/utils/utils_test.go b/pkg/utils/utils_test.go index 2cf8bcd..64b5a14 100644 --- a/pkg/utils/utils_test.go +++ b/pkg/utils/utils_test.go @@ -22,8 +22,8 @@ func TestTrimKey(t *testing.T) { } func TestListFilesOrdered(t *testing.T) { - // test that ListFilesOrdered returns list of files with path - // and they are ordered by their name + // test that ListFilesOrdered returns a list of paths to the files, + // and they are ordered by name testutils.SetUp() defer testutils.Teardown() @@ -39,13 +39,13 @@ func TestListFilesOrdered(t *testing.T) { os.OpenFile(filepath.Join(filesDir, f), os.O_RDONLY|os.O_CREATE, 0600) } - // expFiles with ordered by last modified time and with full path + // expFiles ordered by last modified time and with full path expFiles := []FileInfo{ - FileInfo{ + { Name: ".test/list-files-test/another.sstable", Size: 0, }, - FileInfo{ + { Name: ".test/list-files-test/file.sstable", Size: 0, }, @@ -54,8 +54,8 @@ func TestListFilesOrdered(t *testing.T) { } func TestListFilesOrderedWithoutSuffix(t *testing.T) { - // test that ListFilesOrdered returns list of files with path - // and they are ordered by their name + // test that ListFilesOrdered returns a list of paths to the files + // ordered by name testutils.SetUp() defer testutils.Teardown() @@ -70,17 +70,17 @@ func TestListFilesOrderedWithoutSuffix(t *testing.T) { os.OpenFile(filepath.Join(filesDir, f), os.O_RDONLY|os.O_CREATE, 0600) } - // expFiles with ordered by last modified time and with full path + // expFiles ordered by last modified time and with full path expFiles := []FileInfo{ - FileInfo{ + { Name: ".test/list-files-test/3.sstable", Size: 0, }, - FileInfo{ + { Name: ".test/list-files-test/2.sstable", Size: 0, }, - FileInfo{ + { Name: ".test/list-files-test/1.txt", Size: 0, },