Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add docs for database schema and cedar #779

Merged
merged 1 commit into from
Dec 24, 2024
Merged

add docs for database schema and cedar #779

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
582 changes: 582 additions & 0 deletions aria/contents/docs/architecture/database/index.mdx
View file
Open in desktop

Large diffs are not rendered by default.

77 changes: 77 additions & 0 deletions aria/contents/docs/architecture/policy/index.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
title: Policy
description: This section provides an architectural overview of the Mega policy system.
---

We Use Cedar as our Policy Management, find more at [Cedar](https://www.cedarpolicy.com/)


## Initialization
When initializing Mega’s default directory, the system generates a .mega_cedar.json file, which by default defines resources such as user, repo, and usergroup.

In the Saturn project, there are two Cedar configuration files,

**mega_policies.cedar**: Define principal, action and resource relations.
**mega.cedarschema**: Define entity structure of principal, action and resource.

## Cedar Json File
```json
{
"users": {
"User::\"admin\"": {
"euid": "User::\"admin\"",
"parents": [
"UserGroup::\"admin\""
]
}
},
"repos": {
"Repository::\"/project\"": {
"euid": "Repository::\"/project\"",
"is_private": true,
"admins": "UserGroup::\"admin\"",
"maintainers": "UserGroup::\"matainer\"",
"readers": "UserGroup::\"reader\"",
"parents": []
}
},
"user_groups": {
"UserGroup::\"admin\"": {
"euid": "UserGroup::\"admin\"",
"parents": [
"UserGroup::\"matainer\""
]
},
"UserGroup::\"matainer\"": {
"euid": "UserGroup::\"matainer\"",
"parents": [
"UserGroup::\"reader\""
]
},
"UserGroup::\"reader\"": {
"euid": "UserGroup::\"reader\"",
"parents": []
}
},
"merge_requests": {},
"issues": {}
}
```

## Custom Permissions
When performing permission checks on sub-level files, the configurations of the parent directory are inherited.

If custom permissions are required for a subdirectory, a new .mega_cedar.json file needs to be created in the subdirectory. This file should include the corresponding `user euid`, select the appropriate `user group` in parents field, and modify the `repo path euid`. Once configured, this file should be committed along with the code.

## How to check Permission
Currently, since only the API involves permission checks, the `check_permissions` function is temporarily placed in the mono module. This function can be used to perform permission checks by passing in the Principal, Action, and Resource.

**Principal**: User Gihub Login Name.<br/>
**Action**: Defined Operation Enums to resources.<br/>
**Resource**: Path of Repo.


## Mega Permission Loading Process
For the entire path from the root directory to the target directory, all .mega.json entity files need to be read. These files are then parsed into entities, which are subsequently merged. (This means that the administrators of a parent directory are also administrators of its subdirectories.)

<Outlet path="policy" />
8 changes: 8 additions & 0 deletions aria/lib/routes-config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,14 @@ export const ROUTES: EachRoute[] = [
{ title: "Orion Module", href: "/orion" },
],
},
{
title: "Database",
href: "/database",
},
{
title: "Policy",
href: "/policy",
}
],
},
{
Expand Down
203 changes: 2 additions & 201 deletions ceres/src/api_service/mono_api_service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use std::collections::{HashMap, VecDeque};
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::str::FromStr;
use std::{env, fs};
Expand Down Expand Up @@ -196,61 +196,6 @@ impl ApiHandler for MonoApiService {
}

impl MonoApiService {
pub async fn mr_tree_files(&self, link: &str) -> Result<Vec<PathBuf>, MegaError> {
let storage = self.context.services.mono_storage.clone();
let model = self.context.mr_stg().get_mr(link).await.unwrap();
if let Some(model) = model {
let to_tree_id = storage
.get_commit_by_hash(&model.to_hash)
.await
.unwrap()
.unwrap()
.tree;

let from_tree_id = storage
.get_commit_by_hash(&model.from_hash)
.await
.unwrap()
.unwrap()
.tree;

let mut tree_comparation = TreeComparation {
left_tree: VecDeque::new(),
result_file: vec![],
};
tree_comparation.left_tree.push_back((
SHA1::from_str(&to_tree_id).unwrap(),
Some(SHA1::from_str(&from_tree_id).unwrap()),
PathBuf::from(model.path),
));
while let Some((new_tree, base_tree, mut current_path)) =
tree_comparation.left_tree.pop_front()
{
let new_tree: Tree = storage
.get_tree_by_hash(&new_tree.to_string())
.await
.unwrap()
.unwrap()
.into();

let base_tree = if let Some(base_tree) = base_tree {
let base_tree: Tree = storage
.get_tree_by_hash(&base_tree.to_string())
.await
.unwrap()
.unwrap()
.into();
Some(base_tree)
} else {
None
};
tree_comparation.compare(new_tree, base_tree, &mut current_path);
}
return Ok(tree_comparation.result_file);
}
Err(MegaError::with_message("Can not find related MR by id"))
}

pub async fn merge_mr(&self, mr: &mut MergeRequest) -> Result<(), MegaError> {
let storage = self.context.services.mono_storage.clone();
let refs = storage.get_ref(&mr.path).await.unwrap().unwrap();
Expand Down Expand Up @@ -434,91 +379,9 @@ impl MonoApiService {
}
}

pub struct TreeComparation {
pub left_tree: VecDeque<(SHA1, Option<SHA1>, PathBuf)>,
pub result_file: Vec<PathBuf>,
}

impl TreeComparation {
// fn compare(&mut self, new: Tree, base: Option<Tree>) {
// let new_set: HashSet<_> = new.tree_items.into_iter().collect();
// let base_set = if let Some(tree) = base {
// tree.tree_items.into_iter().collect()
// } else {
// HashSet::new()
// };
// let diff: Vec<_> = new_set.symmetric_difference(&base_set).cloned().collect();
// for item in diff {
// if item.mode == TreeItemMode::Tree {
// let t_id = item.id.to_string();
// if !self.left_tree.contains(&t_id) {
// self.left_tree.push_back(t_id);
// }
// } else {
// self.result_file.push(item.name)
// }
// }
// }

pub fn compare(&mut self, new: Tree, base: Option<Tree>, current_path: &mut PathBuf) {
let mut map_name_to_item = HashMap::new();
let mut map_id_to_item = HashMap::new();

if let Some(tree) = base {
for item in tree.tree_items {
map_name_to_item.insert(item.name.clone(), item.clone());
map_id_to_item.insert(item.id, item.clone());
}
}

for item in new.tree_items {
match (
map_name_to_item.get(&item.name),
map_id_to_item.get(&item.id),
) {
(Some(base_item), _) if base_item.id != item.id => {
self.process_diff(item, Some(base_item.id), current_path);
}
(_, Some(base_item)) if base_item.name != item.name => {
self.process_diff(item, None, current_path);
}
(None, None) => {
self.process_diff(item, None, current_path);
}
_ => {}
}
}
}

fn process_diff(&mut self, item: TreeItem, base_id: Option<SHA1>, current_path: &mut PathBuf) {
if item.mode == TreeItemMode::Tree {
if !self
.left_tree
.contains(&(item.id, base_id, current_path.to_path_buf()))
{
current_path.push(item.name);
self.left_tree
.push_back((item.id, base_id, current_path.to_path_buf()));
current_path.pop();
}
} else {
current_path.push(item.name);
self.result_file.push(current_path.to_path_buf());
current_path.pop();
}
}
}

#[cfg(test)]
mod test {
use std::{collections::VecDeque, path::PathBuf};

use mercury::{
hash::SHA1,
internal::object::tree::{Tree, TreeItem, TreeItemMode},
};

use crate::api_service::mono_api_service::TreeComparation;
use std::path::PathBuf;

#[test]
pub fn test() {
Expand All @@ -530,66 +393,4 @@ mod test {
println!("name: {}, path: {:?}", name, full_path);
}
}

#[test]
fn test_compare_tree() {
let base = Tree::from_tree_items(vec![
TreeItem {
mode: TreeItemMode::Tree,
id: SHA1::new(b"src"),
name: "src".to_string(),
},
TreeItem {
mode: TreeItemMode::Tree,
id: SHA1::new(b"mega"),
name: "mega".to_string(),
},
TreeItem {
mode: TreeItemMode::Blob,
id: SHA1::new(b"delete.txt"),
name: "delete.txt".to_string(),
},
TreeItem {
mode: TreeItemMode::Blob,
id: SHA1::new(b"README.md"),
name: "README.md".to_string(),
},
])
.unwrap();

let new = Tree::from_tree_items(vec![
TreeItem {
mode: TreeItemMode::Tree,
id: SHA1::new(b"src"),
name: "src_new".to_string(),
},
TreeItem {
mode: TreeItemMode::Tree,
id: SHA1::new(b"mega222"),
name: "mega".to_string(),
},
TreeItem {
mode: TreeItemMode::Blob,
id: SHA1::new(b"Cargo.toml"),
name: "Cargo.toml".to_string(),
},
TreeItem {
mode: TreeItemMode::Blob,
id: SHA1::new(b"README.md"),
name: "README.md".to_string(),
},
])
.unwrap();

let mut tree_comparation = TreeComparation {
left_tree: VecDeque::new(),
result_file: vec![],
};
// let mut current_path = Path::new("/");
tree_comparation.compare(new, Some(base), &mut PathBuf::from("/"));
println!(
"files: {:?} left Tree: {:?}",
tree_comparation.result_file, tree_comparation.left_tree
);
}
}
22 changes: 0 additions & 22 deletions docs/database.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,6 @@ Mega's storage structure is mainly divided into the following parts:

Similar to the 'tree' in Git, Mega maintains relationships between files and file names. In the database, Mega independently manages directory information for the current version.

### b-link file*

The purpose of the B-link file is to store file index information, serving as a replacement for blobs in Git. The design of this structure is inspired by the specification of Git LFS, as follows:

```bash
version https://gitmega.dev/spec/v1
blob 3a739f77180d81aa45d9bd11eb6be7098bf1991f
storage_type local_fs
storage_location /tmp/.mega/{{reponame}}/.objects/3a/73/9f77180d81aa45d9bd11eb6be7098bf1991f
```

```bash
version https://gitmega.dev/spec/v1
{{objectType}} {{sha1}}
storage_type {{type}}
storage_locaton {{location}}
```

It includes the following records:
- version: Represents the version information of the structure.
- blob: Points to the hash value of the actual blob.

### Import directory
- The primary purpose of importing directories is to synchronize the original Git repository into the Mega directory. Projects within the import directory are maintained in a **read-only** state, preserving the original commit information.
- Projects pushed to the import directory can have multiple commits.
Expand Down
1 change: 0 additions & 1 deletion mono/src/api/mr/mr_router.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ pub fn routers() -> Router<MonoApiServiceState> {
.route("/mr/:link/merge", post(merge))
.route("/mr/:link/close", post(close_mr))
.route("/mr/:link/reopen", post(reopen_mr))
// .route("/mr/:link/files", get(get_mr_files))
.route("/mr/:link/files-changed", get(get_mr_files_changed))
.route("/mr/:link/comment", post(save_comment))
.route("/mr/comment/:conv_id/delete", post(delete_comment))
Expand Down
Loading