We can fine tune language models like BERT, GPT-3.
Can I fine tune GitHub Copilot model?
I have already looked into examples from https://copilot.github.com/ but cant find the details.
Would really appreciate if someone had fine tuned Github Copilot.
OpenAI API offers the "Davinci Codex" machine learning model with a pay-per-hit subscription, similar to the the non-coding version of the davinci model.
OpenAI should enable the fine-tuning option to Davinci Codex as well. When they do it you will be able to use it via API calls.
After checking that prerequisite, I think you could link the OpenAI API to your local installation of Github Copilot via some code changes, in theory that should be possible.
The first step is probably to have a fork of the copilot VSCode extension that calls the OpenAI Codex API (or an entirely custom extension which inserts text in your code)
Then you would point it to your fine-tuned version of the model. To learn about fine-tuning OpenAI models you should look at their documentation:
https://beta.openai.com/docs/guides/fine-tuning
Note that they have also a openai CLI that allows you to do most of the data loading and fine tuning tasks.
Unfortunately at the moment you can only fine tune the non-coding version of OpenAI models, hope that they will make available Codex soon.
There does not seem to be a client-facing feature allowing you to fine-tune Copilot directly.
Here are two illustration as to why this feature is, for now (Q2 2022) missing.
The Copilot feature page initially included this:
How will GitHub Copilot get better over time?
GitHub Copilot doesn’t actually test the code it suggests, so the code may not even compile or run. GitHub Copilot can only hold a very limited context, so even single source files longer than a few hundred lines are clipped and only the immediately preceding context is used. And GitHub Copilot may suggest old or deprecated uses of libraries and languages. You can use the code anywhere, but you do so at your own risk.
As Tomek Korbak explains on Twitter:
Actually, Copilot's completions will always be optimised for human's liking, not necessarily compiler's liking.
That's because the language model training objective (predicting the next token in text) is great at capturing short-term dependencies (which explains the human feel of generated snippets).
But it struggles to capture long-term, global, semantic properties of generated sequences such as compilability. And there's no easy way of including compilability as a signal for their training.
The standard way -- fine-tuning language models using RL with compilability as a reward -- notoriously leads to catastrophic forgetting: less diverse and less accurate completions.
Tomek references "Energy-Based Models for Code Generation under Compilability Constraints (pdf)"
Our solution (KL-DPG) boosts compilability rate of generated sequences from 55% to 70%.
RL fine-tuning can do better but at a cost of catastrophic forgetting.
Overall, energy-based models (EBMs) turn out to be great at expressing weird, sequence-level constraints that would be super hard as to express as normalised priors for autoregressive language models.
EBMs provide a way of injecting our structured, symbolic knowledge into large language models without breaking them down or sacrificing their uncanny abilities.
The space of further applications in controllable generation is huge.
So not so easy.
Tanishq Mathew Abraham explains in "Coding with GitHub Copilot"
I wonder if the GitHub team might also develop a way of perhaps fine-tuning GitHub Copilot to specific use-cases.
For example, there may be a specific GitHub Copilot models for fastai, JAX, etc. They would be fine-tuned on the source code of of these libraries and codebases that use these libraries.
But making sure that the tool does not provide outdated suggestions would still be a challenge.
I don’t think it would be possible to provide suggestions for a brand-new library that does not have enough codebases using it to train on.
Additionally, for situations like fastai where there are older APIs and newer APIs, when fine-tuning a model, the codebases using the older APIs would have to be filtered out.
No, not at all. There is no storing of GitHub Pilot's model in the client's system and also no access is provided to the model since now they have started charging for their services hence making it more obvious that their project isn't and won't be open-sourced.
I have a standard set of scenarios (about 100 request messages) that need to run against a service.
I want to run this same set against a variety of configurations with possibly different expected outcomes.
Ideally I would be able to say...
Here's my scenarios and default configuration and default results.
Here are other configurations and different expected results. Best if I only need to specify results that differ from the default.
Can I do this in cucumber without having to repeat the requests for each configuration?
I know I won't get exactly what I want, but any way to approximate this?
I have a hard time to see that you are describing a problem Cucumber would be good at supporting.
Behaviour-Driven Development, BDD, is all about communication. Communicating how something should behave in such a way that it is
Understandable by non techies
Thought provoking so you explore and find unknown unknowns
Executable
What you are describing is something I personally would use any testing framework to solve. You wouldn't be using scenarios to discuss a desired behavior for a system. You would use the tool more like a test runner. And using Cucumber like this is possible, but other tooling would solve the problem just as good and with less work.
I am trying to determine which of these two to buy for my work. I have used SIMULINK but not LabVIEW. Is there anyone who has used both and would like to provide some details? My investigation criteria are the user friendliness, availability of libraries and template functions, real-time probing facility, COTS hardware interfacing opportunity, quality of code generation, design for testability (i.e. ease of generating unit/acceptance tests), etc. However, if anyone would like to educate me with more criteria, please do so by all means!
For anyone who does not know about SIMULINK and LabVIEW - These are both Domain-Specific Languages (DSLs) intended for graphical dataflow modelling (and also code generation). These are multi-industrial tools and quite heavily used for engineering design and modelling.
IMPORTANT - I am quite interested to know if SIMULINK and LabVIEW offer real-time probing. For example, I have a model that I want to simulate. If there are variables associated to certain building blocks in that model, could I view them changing as the simulation continues? I know that it is certainly not possible with SIMULINK as it has a step-by-step debugger. I am not aware of anything similar in LabVIEW.
I really have not used LabVIEW and cannot obtain it temporarily as my work internet has got download restrictions and administrative privilege issues. This is the reason why I simply cannot use only NI website to draw conclusions. If there is any white paper available that addresses this issue, I would also love to know :)
UPDATE SINCE LAST POST
I have used MATLAB code generator and will not say that it is the best. However, I hear now that SIMULINK Embedded Coder is the best code generator and almost one of its own kind. Can anyone confirm whether or not this is good for safety critical system design i.e. generating code from safety-critical subsystem models. I know that the Mathworks is constantly trying to close the gap to achieve fully-flexible production-level C/C++ code generation.
I know that an ideal answer would be,"Depending on what you are trying to do, use a bit of both". And interestingly, I think I am heading to that direction. ATEOTD, it is a lot of money and need to be spent "nicely".
Thanks in advance.
I used labVIEW from 1995, and Simulink from 2000. Now I am involved in control system design, and simulation of robotic systems using labVIEW Real Time and automotive ECUs using MATALAB/Simulink/DSPACE .
LabVIEW is focus on measurement systems, and MATLAB/SIMULINK in dynamic simulation, so,
If you run complex simulations, and your work is create/debug complex simulation models of controllers or plants, use Simulink+RealTimeWorkShop+StateFlowChart. LabVIEW has no eficient code generators for dynamic simulation. RTW generates smaller and fastest code.
If your main work is developing systems with controllers and GUI for machines, or you want to deploy the controllers on field, use labVIEW.
If your main work is developing flexible HIL or SIL systems, with a good GUI, you can use VeriStand. Veristand can mix Simulink and LabVIEW code.
And if you have a big budget ( VERY BIG ) and you are working in automotive control prototypes, DSPACE hardware is a very good choice for fast development of automotive ECUS, or OPAL to develope electric power circuits. But only for prototype or HIL testing of controllers.
From the point of view of COTS hardware:
Mathworks don´t manufacture hardware -> Matlab/Simulink support hardware from several vendors.
National Instruments produce/sell hardware->LabVIEW Real Time is focused in support NationalInstruments hardware. There are no COTS full replacement.
I have absolutely no experience with Simulink, so I'll comment only on LV, although a quick read about Simulink on Wikipedia seems to indicate that it's focused mainly on simulation and modelling, which is certainly not the case with LabVIEW.
OK, so first of all, LV is NOT a DSL. While you wouldn't want to use it for any project, it's a general purpose programming language and you should take that into account. I know that NI has a simulation toolkit for LV, which might help you if that's what you're after, but I have absolutely no experience with it. The images I saw of it seemed to indicate that it adds a special kind of diagram to LV for simulation.
Second, LV is not restricted to any kind of hardware. It's a general purpose language, so you can write code which won't use any hardware at all, code which will use or run on NI's hardware or code which will use any hardware (be it through DLL calls, .NET assemblies, RS232, TCP, GPIB or any other option you can think of). There is quite a large collection of LV drivers for various devices and the quality of the driver usually depends on who wrote it.
Third, you can certainly probe in real time in LV. You write your code, just as you would in C or Java, and when you run it, you have several debugging options:
Single stepping. This isn't actually all that common, partially because LV is parallel.
Execution highlighting. This runs the code in slow motion, while showing all the values in the various wires.
Probes, which show you the last value that each wire had, where wires fill the same function that variables do in text based languages. This updates in real time and I assume is what you want.
Retain wire values, which allows you to probe a wire even after data passed through it. This is similar to what you get in text based IDEs with variables. In LV you don't usually have it because wire values are transient, so the value is not kept around unless you explicitly ask for it.
Of course, since you're talking about code, you could also simply write the code to display the values to the screen on a graph or a numeric indicator or to log them to a file, so there should be no need for actual probing. You could also add analysis code, etc.
Fourth, you could try downloading and running LV in a fully functional evaluation mode. If I remember correctly, NI currently gives you 7 days and then 45 days if you register on their site. If you can't do that on a work computer, you could try at home. If your problem is only with downloading, you could try contacting your local NI office and asking them to send you a DVD.
Note that I don't really know anything about modelling and simulation, so I have no idea what kind of code you would actually have to write in order to do what you want. I assume that if NI has a special module for it, then it's not something that you can completely cover in regular code (at least not if you want the original notation), but I would say that if you could write the code that does what you want in C, there's no reason you shouldn't be able to write it in LV (assuming, of course, that you know how to write code in LV).
A lot of the best answer would have to depend on your ultimate design requirements. Are you developing a product? If so, in what stage of development are you? Or are you doing research?
I recently did a comparison just as you are doing. I know LV, but was wanting to move towards a more hardware-scalable option, since NI HW is very expensive in volume. That is, my company was wanting to move towards a product. What LV and NI HW give you is flexibility. You can change code very quickly compared to C. On the other hand, LV does not run on nearly as many different HW platforms as C. So I wanted to find an inexpensive platform that would work well for real-time control and data acquisition, such that if we wanted to sell a product for, say, $30k, our controller wouldn't be costing $15k of that. We ended up with Diamond Systems Linux SBC's. Interestingly, Simulink ended up using the most expensive hardware! It did have a lot of flexibility, and could generate code, as well as model plants and controllers. But then, LV can do that as well.
As Yair wrote, LV has plenty of good debugging tools. One of the more interesting tools that is not so well known is the Suspend when Called option for a SubVI. This allows you to play with the inputs and outputs of a SubVI as much as you want while execution is paused.
MATLAB and Simulink are the defacto standard for control system design and simulation. Simulink controller models can be used for offline simulation in conjunction with plant models, all the way to realtime implementation on embedded targets. It is a general simulation framework with extensive built-in libraries as well as a la carte special purpose libraries, and can be extended through creation of custom blocks (S-function blocks) in C and other languages. It includes the ability to display values in graphs, numeric displays, gages, etc. while a nonrealtime simulation is taking place. Realtime target support from The Mathworks includes x86 (xPC Target) and several embedded targets (MPC555, etc.), and there is 3rd party support for other targets. The aforementioned dSPACE provides complete prototyping controllers including support for their quite powerful hardware. xPC Target includes support for a plethora of COTS PC data acquisition cards. Realtime target support includes GUI elements such as graphs, numeric displays gages, etc.
As I understand it (I have never really used it in anger), LabView only supports NI hardware, and is more hardware-oriented. Simulink supports hardware from multiple vendors, be it for data acquisition, or real-time implementation, but it may require a bit more work for the user to interface to his or her own hardware (less plug & play than LabView). On the other hand, Simulink provides tools to support the whole model-based design process, from modelling & simulation, control design, verification & validation, code generation, hardware-in-the-loop, etc...
Disclaimer: I used to work for MathWorks.
You guys may really be interested in Control Design adn Simulation Module for LabVIEW. It does a lot of simulations and in the future may be competitive to Simulink. I'm not a control engineer but I use it sometimes for simple testing and I'm glad that I don't have to learn Simulink from the beginning to do some work since I'm familiar with LabVIEW philosophy.
We need to visualize BP (business process) into BPMN, but NOT by hands using modeler. We need to do it automatically in crm-web-based system written on PHP. I have input data (etc. array, xml, not care...(but not BPEL)), then I need to process it into nice BPMN graph (using SVG).
We have first nice-looking realization of it. We use matrix to draw: several times goes through matrix and optimize graph each time, no no, it working fast, but it not agile, hard to rebuilt, upgrade, add new features... We made this algorithm by ourselves (I mean we didn't find it in google or books). Problem is that we couldn't find any algorithms in the internet. I suppose we don't know correct keywords to do it. Every try returned us to BPEL vis. from BPMN, "Data flow vis." returned modelers...
Please help us to find some algorithms, or give correct keywords to find out information.
Think you're probably looking for "graph layout algorithms". The only library I'm aware of that can (I think) generate BPMN directly is the yFiles library from yWorks. It's not free. They do however offer a free application using the library that does auto-layout. Perhaps you could do some prototyping with that.
If that's not applicable, there are several other options. I'm not aware any of these can generate BPMN symbols directly; you'd have to construct the symbols. However all will auto-layout graphs according to various algorithms. Also all open source/free.
graphviz. Written in C. Quite old now but well used, stable and scalable.
tulip. Newer than graphviz. Haven't used it but heard good things about flexibility and scalability.
see also this post for javascript based options.
There are many more, just google for graph layout algorithms / libraries.
hth.
Our company plans to auto-generate our projects from the domain area up-to the presentation layer so that we can mass produce software. The idea is we can produce a project within 24 hours. I think this is possible but that's not my concern.
What are the ramifications of such plan? I just think that the quality of software produced from such grandeur idea is below good quality. First, clients have varying requirements. Assuming we can standardize what's common among them, there are still requirements that will be beyond our original template.
Second, how can such software be reliable if not fully tested? Does a 24 hour period can cover a full unit/integration/other types of test?
At the end, it appears we won't be able to hit the 24 hour target thereby defeating our original purpose.
I just believe it's better to build quality software than mass producing them. How would I tell my boss that their idea is wrong?
Sorry, but I don't think this is sensible.
In order to build a system that can auto-generate any kind of software that would fill any kind of requirement, you will kind of have to implement more then the softwares you plan to generate.
Auto-generated code is great, when you have some repetitive tasks, information, or components, that are similar enough to enable you to make a one time effort to generate all repetitions.
But trying to write a system to produce any kind of project is not feasible. In order to meet a wide enough range of supported projects, your system will have to allow a very wide set of capabilities to describe project configuration and behavior, and the time required to describe each project's behavior will not necessarily be shorter than the time it would have taken to implement the project in the first place. You will just end up developing a development environment, and implementing projects in your own language.
So, instead, why not just take an existing development environment that is already available? Like Visual Studio?
Maybe you should have a big library reprository and stuff everything that could be reusable in it. This way, the software you have to write would be very small. There could be documentation templates associated with the library and you would just have to C&P them.
However, it takes time to build such a library.
Or do it this way.