running locally question

[CristiZZzz27]

how can i set the input/output file path? I dont see it in the quick start, thanks.

[ChenZiHong-Gavin]

In scripts/generate.sh, the default command is:

python3 -m graphgen.generate --config_file graphgen/configs/graphgen_config.yaml --output_dir cache/

where output_dir is the output directory.
In the config_file, you can set the input file path by specifying input_file.

[CristiZZzz27]

In scripts/generate.sh, the default command is:

python3 -m graphgen.generate --config_file graphgen/configs/graphgen_config.yaml --output_dir cache/

where output_dir is the output directory. In the config_file, you can set the input file path by specifying input_file.

After running the code, I’ve got a few questions:

How do I control the number of QA pairs it generates?
No matter how much content I put in the jsonl file, it always ends up making just 5 QA pairs.

If the knowledge content changes next time, do I need to delete the cache and run it again?
Because it seems like it’s still using old data from graph.graphml.

Are there any key parameters I should pay attention to or adjust each time I run it?

My API has the logprobs feature, but I'm still seeing all the loss values are the same. Do I need to manually enable hard case mining?

[ChenZiHong-Gavin]

In the config file, we have the following parameters:

traverse_strategy:
  qa_form: aggregated
  bidirectional: true
  edge_sampling: max_loss
  expand_method: max_width
  isolated_node_strategy: ignore
  max_depth: 1
  max_extra_edges: 2
  max_tokens: 256
  loss_strategy: only_edge

These parameters are used to extract subgraphs from the complete knowledge graph. Subgraph extraction starts with an edge in the knowledge graph and expands outward through associations. The bidirectional parameter determines the direction of candidate edge selection; if set to true, it means expanding in both directions. If expand_method is set to max_width, it means limiting the maximum number of nodes within a subgraph during expansion, controlled by max_extra_edges. If expand_method is set to max_tokens, it means limiting the input text length based on the token count of all nodes and edges in the current subgraph. Meanwhile, max_depth restricts the maximum expansion depth in a single direction.

You can refer to the documentation: https://graphgen-cookbook.readthedocs.io/en/latest/parameters.html#expand-method.

Regarding the knowledge content, if you are using entirely new data instead of incremental data, you need to delete the cache or change the output directory. This is because GraphGen only maintains a global knowledge graph.

As for the logprobs feature, you may want to check the implementation of the generate_topk_per_token method in the graphgen/models/llm/openai_model.py file and print the return value of the LLM to see whether the issue lies with the interface or the code implementation.

[CristiZZzz27]

In the config file, we have the following parameters:

traverse_strategy:
  qa_form: aggregated
  bidirectional: true
  edge_sampling: max_loss
  expand_method: max_width
  isolated_node_strategy: ignore
  max_depth: 1
  max_extra_edges: 2
  max_tokens: 256
  loss_strategy: only_edge

These parameters are used to extract subgraphs from the complete knowledge graph. Subgraph extraction starts with an edge in the knowledge graph and expands outward through associations. The bidirectional parameter determines the direction of candidate edge selection; if set to true, it means expanding in both directions. If expand_method is set to max_width, it means limiting the maximum number of nodes within a subgraph during expansion, controlled by max_extra_edges. If expand_method is set to max_tokens, it means limiting the input text length based on the token count of all nodes and edges in the current subgraph. Meanwhile, max_depth restricts the maximum expansion depth in a single direction.

You can refer to the documentation: https://graphgen-cookbook.readthedocs.io/en/latest/parameters.html#expand-method.

Regarding the knowledge content, if you are using entirely new data instead of incremental data, you need to delete the cache or change the output directory. This is because GraphGen only maintains a global knowledge graph.

As for the logprobs feature, you may want to check the implementation of the generate_topk_per_token method in the graphgen/models/llm/openai_model.py file and print the return value of the LLM to see whether the issue lies with the interface or the code implementation.

I find that the code

if (not re_judge) and "loss" in edge_data and edge_data["loss"] is not None:
logger.info("Edge %s -> %s already judged, loss: %s, skip", source_id, target_id, edge_data["loss"])
return source_id, target_id, edge_data

there are default loss = -math.log(0.1), so "loss = yes_no_loss_entropy(judgements, gts)" will never be used, so the loss is fixed. Is it designed on purpose?

[ChenZiHong-Gavin]

This is because you already have comprehension loss value in the knowledge graph (You may check graph.graphml).
You can set re_judge to True to get new loss.

This is to avoid having to regenerate the loss each time. If there is a better way to implement this, feel free to submit a PR.